How does the message receiving mechanism work in Kafka consumer? Where is the offset stored?

A Kafka consumer receives messages from a topic by subscribing to specific partitions. It reads messages sequentially, starting from a certain offset — the position within the partition.

Working mechanism:

• The consumer requests messages from the broker, starting from the current offset.
• After processing messages, the consumer can commit the offset to continue reading from the last processed message upon restart or failure.

Where the offset is stored:

• In Kafka, the offset is usually stored in a special internal topic __consumer_offsets.
• It can also be configured to be stored in external systems (e.g., a database) or managed manually.

This allows consumers to be resilient to failures and not lose their reading position.