How to ensure exactly-once semantics in Kafka? How is producer idempotency technically ensured?

Exactly-once semantics in Apache Kafka are achieved through producer idempotence and transactions.

Producer idempotence guarantees that resending the same message will not result in duplication in the topic.

Technically, idempotence is implemented as follows:

• During initialization, the producer receives a unique producerId and maintains a sequenceNumber counter for each partition.
• Each message is sent with an incremented sequenceNumber.
• The broker checks the producerId and sequenceNumber and discards duplicates (messages with an already processed number).

To enable producer idempotence in Kafka, set the parameter enable.idempotence=true.

Example of configuring a producer in Java:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // Enable idempotence

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

For full exactly-once semantics during message processing (e.g., reading, processing, and writing to another topic), a transactional producer with parameters transactional.id and methods initTransactions(), beginTransaction(), commitTransaction(), abortTransaction() is used.

Thus, producer idempotence is a mechanism to control message uniqueness at the broker level using producerId and sequenceNumber, preventing duplication on retries.