When can we call commit on a Producer? How to avoid duplicates when manually committing a batch?

In Kafka Producer, the commit method is not called because offset commits are handled by the Consumer. If we're talking about the Consumer, then commit is called after successful message processing to confirm that messages have been read and processed.

To avoid duplicates when manually committing a batch:

• Process the entire batch of messages atomically.
• After successfully processing all messages in the batch, call commit.
• If processing is interrupted, do not call commit so that the same messages can be processed again upon restart.

Example for Kafka Consumer in Python using the kafka-python library:

from kafka import KafkaConsumer

consumer = KafkaConsumer('topic', enable_auto_commit=False)

for messages in consumer:
    try:
        for message in messages:
            process(message)  # Your processing logic
        consumer.commit()  # Manual commit after successful batch processing
    except Exception:
        # Processing error, do not commit to avoid data loss
        pass

Thus, commit is called only after the entire batch is successfully processed, minimizing message duplication.