How does Kafka work with multiple pods in one consumer group? If the first reads a message, will the second see it?

In Kafka, a consumer group is a group of consumers that jointly read data from topics. Each partition of a topic can be assigned to only one consumer in the group. If you have multiple pods (consumer instances) in one consumer group, Kafka distributes the partitions among them.

This means that if the first pod has read a message from its partition, the second pod, which reads from another partition, will not see this message because each message is delivered to only one consumer in the group. Thus, messages are not duplicated among pods within the same consumer group.

Example: if a topic has 4 partitions and there are 2 pods in the group, each pod will read from 2 partitions. Messages from partitions assigned to the first pod will not go to the second, and vice versa.