There is a topic with 5 partitions. How many consumers is optimal to create? What happens with 4 consumers? What happens with 6 consumers on 5 partitions?

If a Kafka topic has 5 partitions, it is optimal to create exactly 5 consumers in one group so that each consumer processes its own partition without competition.

• 4 consumers for 5 partitions: one partition will be left without a consumer, meaning one consumer will read two partitions. This is acceptable, but the load on consumers will be uneven.

• 6 consumers for 5 partitions: one consumer will be left without a partition and will be idle, as a partition cannot be divided among multiple consumers in one group.

Summary:

• The number of consumers in a group should not exceed the number of partitions, otherwise some consumers will be idle.
• Fewer consumers than partitions lead to some consumers processing multiple partitions.

Example in Go using sarama:

// Create a group with 5 consumers for a topic with 5 partitions
// Each consumer will read its own partition