What delivery guarantee will we use in Kafka?

In Kafka, the 'at least once' delivery guarantee is usually used. This means that a message will be delivered to the consumer at least once, but duplicates may occur in case of failures. To improve reliability and prevent data loss, Kafka uses acknowledgments (acks), where you can set the acknowledgment level:

• acks=0 — the producer does not wait for acknowledgment (least reliable)
• acks=1 — acknowledgment from the partition leader
• acks=all (or -1) — acknowledgment from all replicas (maximum reliability)

To ensure 'exactly once' delivery, additional logic on the consumer side and the use of transactions in Kafka are required, which is more complex and not always justified.

Example of configuring a producer in Go with 'at least once' delivery guarantee:

producer, err := sarama.NewSyncProducer(brokers, &sarama.Config{
    Producer: sarama.ProducerConfig{
        RequiredAcks: sarama.WaitForAll, // acks=all
        Retry:       sarama.NewRetry(5),
    },
})