What is a consumer group in Kafka? What are partitions and why are they needed?

In Kafka, a consumer group is a group of consumers that jointly read data from a topic. Each consumer in the group receives a unique set of partitions to process, which allows scaling the reading and load balancing.

Partitions are logical divisions of a topic that enable parallel writing and reading of messages. Each partition stores an ordered stream of messages.

Why partitions are needed:

• They allow scaling performance through parallel processing.
• They ensure message order within a partition.
• They enable consumer groups to distribute load among consumers.

Example: if a topic has 4 partitions and there are 2 consumers in the group, each will get 2 partitions to read from.