Sobes.tech
Senior

There is a topic with 10 partitions. How many containers (consumers) are needed to read all partitions?

sobes.tech AI

Answer from AI

To read all 10 partitions of a Kafka topic, at least 10 consumers are needed if each consumer runs in a separate container and each consumer reads only one partition.

However, if consumers are grouped into a consumer group, then the number of consumers can be less than or equal to the number of partitions. In this case, partitions are distributed among consumers within the group, but a partition cannot be read simultaneously by two consumers from the same group.

In summary:

  • Minimum 1 consumer (if one consumer reads all partitions sequentially).
  • Ideally — no more than 10 consumers to avoid idle consumers.

If the task is to read all partitions in parallel, then exactly 10 consumers are needed (in 10 containers), so that each processes its own partition.