Гарантии доставки в Kafka: at most once, at least once, exactly once. Как обеспечить exactly once?

Kafka provides three main message delivery guarantees:

• At most once — messages are delivered no more than once, with possible message loss.
• At least once — messages are delivered at least once, with possible duplicates.
• Exactly once — messages are delivered exactly once, with no loss or duplicates.

To ensure exactly once delivery in Kafka, transactions and idempotent producers are used. The main steps are:

1. Enable idempotence on the producer (enable.idempotence=true) to avoid duplicates on retries.
2. Use transactions (initTransactions(), beginTransaction(), commitTransaction(), abortTransaction()) to group message sends and offset commits.
3. Consumers should use the read_committed mode to read only committed transactions.

Example in Go using the confluent-kafka-go library:

producer, _ := kafka.NewProducer(&kafka.ConfigMap{
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": true,
    "transactional.id": "my-transactional-id",
})

producer.InitTransactions(nil)
producer.BeginTransaction()

// Sending messages
producer.Produce(&kafka.Message{TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny}, Value: []byte("message")}, nil)

// Commit transaction
producer.CommitTransaction(nil)

Thus, transactions guarantee that either all operations within a transaction are applied or none, ensuring exactly once semantics.