How is message processing uniqueness ensured in Kafka to avoid processing the same message more than once?

In Kafka, message processing uniqueness is achieved through a combination of offset and the commit mechanism for offsets. Each consumer reads messages from partitions in order, and Kafka assigns a unique offset to each message within a partition.

To avoid reprocessing the same message, the consumer must:

• Save (commit) the offset after successful processing of the message.
• Upon restart or failure, the consumer starts reading from the last committed offset.

Thus, if the offset is committed, messages with smaller offsets will not be processed again.

Additionally, to ensure idempotency, unique message identifiers or external duplicate control mechanisms can be used on the consumer side.

Example of committing offset in Java Kafka Consumer:

consumer.subscribe(Collections.singletonList("topic"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        // process message
    }
    consumer.commitSync(); // commit offset after processing
}