Topic with 3 partitions. Case 1: 2 consumers. Case 2: 5 consumers. How are messages distributed?

In Kafka, a topic with 3 partitions distributes messages across partitions, and consumers in a group receive partitions for processing.

Case 1: 2 consumers, 3 partitions

• Partitions are distributed approximately evenly among consumers.
• One consumer will get 2 partitions, and the other will get 1.
• For example:
  • Consumer A: partitions 0 and 1
  • Consumer B: partition 2

Case 2: 5 consumers, 3 partitions

• There are more consumers than partitions.
• Each partition can only be assigned to one consumer in the group.
• Therefore, only 3 consumers will receive data (one partition each), and the remaining 2 will be idle.

Thus, the number of active consumers in a group cannot exceed the number of partitions in the topic. For effective scaling, the number of partitions should be at least equal to the number of consumers.

This is important to consider when designing a system for balanced load and parallel message processing.