Did you have a dynamic number of Kafka consumers?

Yes, a dynamic number of consumers in Kafka is a common practice for scaling message processing. The number of consumers usually changes depending on the load: when data volume increases, new consumers are added; when it decreases, their number is reduced.

It is important to remember that the maximum number of active consumers in a group is limited by the number of partitions in the topic, as one partition can only be processed by one consumer in the group. For dynamic scaling, orchestrators (such as Kubernetes) or auto-scaling systems are often used, which start or stop consumers based on load metrics.

An example of dynamic change in the number of consumers in Kubernetes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-consumer
spec:
  replicas: 3 # can be changed dynamically
  template:
    spec:
      containers:
      - name: consumer
        image: my-kafka-consumer:latest

Auto-scaling can change the replicas based on CPU load or other metrics.