There is a task: atomically write an event to a broker (Kafka/RabbitMQ) and to a database. How to ensure transactional integrity?

Ensuring transactional integrity when recording an event simultaneously in a message broker (Kafka/RabbitMQ) and a database is a non-trivial task, as these are two different systems with different transactional models.

Possible approaches:

1. Two-Phase Commit (2PC) — rarely used due to complexity and lack of support in brokers.

2. Outbox Pattern — within a single transaction, record the event in a special table (outbox) in the database along with the main operation. Then, a separate process reads from this table and sends events to the broker. This guarantees that the event is not lost and not sent twice.

3. Kafka Transactions (for Kafka) — Kafka supports transactions, allowing atomic sending of multiple messages. However, this does not cover the database, so it is often combined with the outbox pattern.

4. Idempotency and Compensation — if atomicity cannot be guaranteed, make operations idempotent and implement compensation logic in case of errors.

Example of Outbox Pattern:

using (var transaction = dbContext.Database.BeginTransaction())
{
    // Write to main table
    dbContext.Events.Add(newEvent);
    // Write to outbox
    dbContext.Outbox.Add(new OutboxMessage { Payload = Serialize(eventData) });
    dbContext.SaveChanges();
    transaction.Commit();
}

// A separate service reads from the Outbox and sends to the broker

Thus, transactional integrity is achieved at the database level, and delivery to the broker is asynchronous but reliable.