How does manual offset commit affect consumer lag?

Manual offset commit in Kafka consumer means that the application independently records up to which message it has successfully processed data from the topic.

Impact on consumer lag:

• Consumer lag is the difference between the last available offset in a partition and the offset that the consumer has already processed and committed.
• If the offset commit is done manually and with delay, the lag will increase, as Kafka considers that the consumer has not yet processed new messages.
• Timely manual commits reduce lag, as they record processing progress.

Thus, manual commit provides more control over when a message is considered processed, but requires careful management to avoid lag accumulation and message loss in case of failures.

Example in Java:

consumer.commitSync(Collections.singletonMap(new TopicPartition(topic, partition), new OffsetAndMetadata(offset + 1)));

Here, offset + 1 is the next offset that the consumer will read, meaning all messages up to and including offset are considered processed.