What is offset in Kafka? How does the offset happen? When to use manual commit vs automatic?

In Kafka, offset is the sequential number of a message in a topic partition, indicating the consumer's position.

How offset is moved: The consumer reads messages sequentially, increasing the offset. Kafka does not delete messages immediately but retains them for a certain period, allowing consumers to read from the desired offset.

Manual commit vs automatic:

• Automatic commit — the client periodically saves the offset in Kafka. Convenient if losing a few messages is not critical. The downside is potential loss or duplication in case of failures.

• Manual commit — the application decides when to save the offset, usually after successfully processing a message. Ensures that messages are not missed or processed twice.

Use manual commit if precise processing and data consistency are important. Automatic is suitable for simple cases with high performance and acceptable data loss.