In a microservices architecture, each service has its own database. How does the Outbox pattern work in this context if Kafka has not yet received the message?

The Outbox pattern is used to ensure atomicity between writing data to a microservice's database and publishing events to Kafka, when each service has its own database.

The idea is that when the service's state changes, both the main business data and the event (message) are written in a special outbox table within the same database transaction. This way, the event recording and the business operation are atomic — either both succeed or both are rolled back.

A separate process (or thread) periodically reads new entries from the outbox table and sends them to Kafka. If sending fails, the record remains in the table, and the attempt is retried later. After successful sending, the record is marked as sent or deleted.

Thus, even if Kafka is temporarily unavailable, data will not be lost, and the service will not violate consistency, as the event is stored locally and will be sent later.

An example of a simplified scheme:

// In a transaction
func SaveOrderAndOutbox(db *sql.DB, order Order, event Event) error {
    tx, _ := db.Begin()
    defer tx.Rollback()

    // Save the order
    _, err := tx.Exec("INSERT INTO orders (...) VALUES (...)", ...)
    if err != nil {
        return err
    }

    // Write event to outbox
    _, err = tx.Exec("INSERT INTO outbox (event_type, payload, processed) VALUES (?, ?, false)", event.Type, event.Payload)
    if err != nil {
        return err
    }

    return tx.Commit()
}

// A separate worker reads from outbox and sends to Kafka
func OutboxWorker(db *sql.DB, kafkaProducer KafkaProducer) {
    for {
        rows, _ := db.Query("SELECT id, event_type, payload FROM outbox WHERE processed = false")
        for rows.Next() {
            var id int
            var eventType, payload string
            rows.Scan(&id, &eventType, &payload)

            err := kafkaProducer.Send(eventType, payload)
            if err == nil {
                db.Exec("UPDATE outbox SET processed = true WHERE id = ?", id)
            }
        }
        time.Sleep(time.Second * 5)
    }
}

This ensures reliable message delivery and data consistency between services.