What mechanisms in Kafka ensure delivery guarantees (at-least-once, exactly-once)?

In Kafka, message delivery guarantees are implemented through the following mechanisms:

1. At-least-once:

   • Automatic offset commits after message processing by the consumer. If the offset is committed after processing, the consumer may receive duplicates in case of failure but will not miss messages.
   • Partition replication across multiple brokers with acknowledgment (acks) to ensure fault tolerance.
   • Setting acks=all when sending messages with the producer to guarantee writing to all replicas.

2. Exactly-once:

   • Using Kafka transactions: producers can send messages within transactions that are either fully committed or rolled back.
   • Idempotent producer: assigns unique identifiers to messages, preventing duplication at the broker during retries.
   • Consumer-side transaction handling with Kafka Streams or support for transactions in the consumer for atomic commit of offsets and data processing.

Thus, the combination of an idempotent producer, transactions, and offset management allows implementing exactly-once semantics, while basic replication and offset acknowledgment provide at-least-once delivery.