How is the processing of different messages organized when using three Consumers in Kafka?

In Kafka, when using three Consumers to process messages, a Consumer Group is usually created that includes these three Consumers. Kafka distributes the topic partitions among Consumers within the group, ensuring that each message from a partition is processed by only one Consumer.

Thus, if the topic has, for example, 3 partitions, each Consumer will read messages from its own partition. If there are fewer partitions than Consumers, some Consumers will be idle. If there are more partitions than Consumers, some Consumers will read from multiple partitions.

This allows parallel processing of messages while avoiding duplicate processing.

Example in Python using the kafka-python library:

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'my_topic',
    group_id='my_group',
    bootstrap_servers=['localhost:9092']
)

for message in consumer:
    print(f"Received message: {message.value.decode('utf-8')}")

If you run three such Consumers with the same group_id, Kafka will distribute the partitions among them.