Is it possible to configure so that each instance of the microservice reads all messages from the topic? And how to distribute processing among pods in Kubernetes?

In Kafka (or similar systems), by default, each instance of a microservice that belongs to a consumer group receives a portion of messages from a topic — messages are distributed among pods to avoid duplicate processing.

If you want each instance to receive all messages from the topic, each instance must have its own unique consumer group. Then, each instance will read all messages completely.

However, this is not scalable for load processing, as each instance processes the entire stream.

To distribute processing among pods in Kubernetes, typically:

• All pods are combined into one consumer group.
• Kafka distributes the topic partitions among pods, ensuring each message is processed by exactly one pod.

Thus, scaling is achieved by increasing the number of partitions and the number of pods in the group.

Example of configuring a consumer group in Java with Kafka:

Properties props = new Properties();
props.put("group.id", "my-consumer-group"); // same group.id for all pods
// ... other settings
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("my-topic"));

If you want each instance to read all messages, set a unique group.id for each:

props.put("group.id", UUID.randomUUID().toString());

But this will lead to duplicate processing.

In Kubernetes, to manage the number of pods and their scaling, a Deployment with the desired number of replicas is used.