How does Kafka work with multiple application instances? What is a consumer group, partitions, rebalancing?

Kafka is a distributed messaging platform that efficiently works with multiple application instances thanks to the concept of consumer groups and partitions.

Consumer group is a group of consumers (application instances) that jointly read data from a topic. Each instance in the group receives a unique set of partitions to process, ensuring parallelism and scalability.

Partitions are logical divisions of a topic that allow distributing data and load among consumers. Each partition is an ordered log of messages.

When multiple application instances join a consumer group, Kafka distributes partitions among them so that each instance processes a unique set of partitions.

Rebalancing occurs when an instance is added or removed from the consumer group, or when the number of partitions changes. Kafka redistributes partitions among active consumers to maintain load balance.

Example: if a topic has 4 partitions and there are 2 instances in the group, each will get 2 partitions. If a third instance is added, rebalancing will redistribute the partitions, for example, 2,1,1.

This allows scaling message processing and ensuring fault tolerance.