How to guarantee atomicity of writing to a database and sending an event to Kafka (Transactional Outbox pattern)?

The Transactional Outbox pattern addresses the problem of guaranteed event delivery in Kafka along with atomic data writing to the database. The main idea is not to send the event directly to Kafka within a transaction, but to write it to a special "outbox" table in the same transaction as the business data.

How it works:

1. Within a single database transaction, write the business data and add a record to the outbox table with the event.
2. After the transaction commits successfully, a separate process (e.g., a dedicated service or background job) reads new records from the outbox and sends them to Kafka.
3. After successful sending, the event is marked as sent or removed from the outbox.

This achieves atomicity: either both the data and the event are written together, or nothing is written. The sending to Kafka occurs asynchronously, but it is guaranteed that the event will not be lost.

Example in Java using Spring and JPA:

@Entity
public class OutboxEvent {
    @Id
    private UUID id;
    private String aggregateType;
    private String aggregateId;
    private String type;
    private String payload;
    private boolean sent;
    // getters and setters
}

@Transactional
public void saveBusinessDataAndEvent(BusinessEntity entity, OutboxEvent event) {
    businessRepository.save(entity);
    outboxRepository.save(event);
}

// Separate service for sending
public void publishOutboxEvents() {
    List<OutboxEvent> events = outboxRepository.findUnsentEvents();
    for (OutboxEvent event : events) {
        kafkaTemplate.send(event.getType(), event.getPayload());
        event.setSent(true);
        outboxRepository.save(event);
    }
}

Thus, we avoid issues with distributed transactions between the database and Kafka, ensuring reliable event delivery.