Sobes.tech
Junior — Middle

How does Kafka handle situations when a message was not successfully delivered to the consumer?

sobes.tech AI

Answer from AI

In Apache Kafka, message delivery to the consumer is guaranteed through acknowledgments and offset management. If a message was not successfully processed by the consumer:

  • The consumer does not acknowledge (commit) the offset associated with that message.
  • Upon restart or re-reading the topic, the consumer will be able to read this message again, as the offset was not advanced.

Kafka itself does not track the success of message processing on the consumer side — this is the client's responsibility. Consumers typically use manual offset management to confirm processing only after successful handling of the message.

If a message is not processed, it remains available for re-reading. Retry mechanisms can be configured, and Dead Letter Queues (DLQ) can be used for messages that fail to be processed after several attempts.

Example in Java with manual offset commit:

ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
    try {
        process(record.value());
        consumer.commitSync(Collections.singletonMap(new TopicPartition(record.topic(), record.partition()),
            new OffsetAndMetadata(record.offset() + 1)));
    } catch (Exception e) {
        // Error handling, message is not acknowledged and will be read again
    }
}