What is a consumer group in Kafka?

A consumer group in Kafka is a group of consumers that jointly read data from topics. Each consumer in the group receives a unique set of partitions to process, ensuring parallelism and scalability.

Key points:

• All consumers in the group have the same group.id.
• Topic partitions are distributed among the consumers in the group to avoid duplicate processing.
• If a consumer fails, its partitions are redistributed among the remaining consumers.

Example: if a topic has 4 partitions and there are 2 consumers in the group, each consumer will read data from 2 partitions. This allows parallel data processing and guarantees that each message is processed by exactly one consumer in the group.