What is offset in Kafka?

In Kafka, offset is a numerical index that indicates the position of a message within a topic partition. Each new message written to a partition receives a unique offset, which increases by 1.

Offsets are used by consumers to track which messages they have already read. This allows the consumer to resume reading from the correct position after a restart or failure.

Example:

• A topic partition has messages with offsets from 0 to N.
• The consumer has read up to offset 100.
• On the next start, it will begin reading from offset 101 to avoid missing new data and re-reading old messages.

Thus, offset is a key mechanism for ensuring sequential and reliable message consumption in Kafka.