Sobes.tech
Senior

Why were 60 consumers used for 30 partitions?

sobes.tech AI

Answer from AI

Using 60 consumers for 30 partitions in Kafka can be related to several reasons:

  1. Increasing parallelism: Although you have 30 partitions, you can run more consumers to ensure redundancy and load balancing. An excess number of consumers can help in case some of them fail.

  2. Redundancy and fault tolerance: Additional consumers can be ready to take over processing of partitions if the current consumers fail.

  3. Different consumer groups: It is possible that 60 consumers are divided into several groups, each reading from the same 30 partitions but for different purposes (e.g., different services or processing types).

  4. Incorrect configuration or preparation for scaling: Sometimes, the number of consumers exceeds the number of partitions during the preparation stage for increasing partitions.

It is important to remember that in Kafka, a single partition can only be processed by one consumer from one group at a time, so an excess number of consumers in one group does not increase throughput but can be used for other purposes.

Why were 60 consumers used for 30 partitions? — Java - sobes.tech