Why is a consumer group needed in Kafka?

A consumer group in Kafka is needed to organize parallel and scalable message consumption from topics.

Each consumer group is a set of consumers that jointly read data from a topic. Kafka guarantees that each partition of a topic will be processed by only one consumer within the group, which allows load distribution and balancing.

Advantages of consumer groups:

• Scalability: more consumers can be added to the group to increase processing speed.
• Reliability: if one consumer fails, another will take over its partitions.
• Guarantee that each message will be processed by exactly one consumer in the group.

Example:

Topic with 4 partitions
Consumer group with 2 consumers
Each consumer reads 2 partitions