What happens during Kafka consumers rebalancing — what can be lost?

When rebalancing consumers in Kafka, partitions are redistributed among them. This is necessary, for example, when adding or removing consumers from a group.

What can go wrong and what might be lost:

• Message loss: If a consumer has not committed the offset of the last read messages before rebalancing, these messages may be read again or skipped during redistribution.

• Duplicate processing: Due to re-reading messages after rebalancing, processing may be duplicated if the application is not idempotent.

• Processing pause: During rebalancing, consumers pause reading, which can lead to delays in message processing.

• Offset loss: If offsets are not committed correctly (for example, using automatic commits with large intervals), after rebalancing, the consumer may start reading from an earlier offset, leading to reprocessing.

Therefore, to minimize losses and duplication, it is important to properly configure offset commits and handle messages in an idempotent manner.