How does a Consumer in Kafka store information about its position in the data stream?

In Apache Kafka, the Consumer saves its position in the data stream using offsets. An offset is the sequential number of a message within a topic partition.

The Consumer periodically commits (saves) its current offset so that upon restart or failure, it can resume reading from the correct position. The commit can be automatic (auto-commit) or manual.

In Kafka clients for Go (e.g., sarama), the offset is usually stored in a special topic __consumer_offsets, managed by Kafka itself. This allows multiple consumers in a group to coordinate reading and ensures that each message is processed exactly once.

Example of manual offset commit in Go using sarama:

partitionConsumer, _ := consumer.ConsumePartition(topic, partition, sarama.OffsetNewest)

for msg := range partitionConsumer.Messages() {
    // process message
    fmt.Println(string(msg.Value))
    // commit offset
    consumer.MarkOffset(msg, "")
}

Thus, the consumer's position in the stream is the last committed offset, stored in Kafka, which allows resuming reading from the correct place.