Why is a Consumer Group needed? How to do broadcast in Kafka?

A Consumer Group in Kafka is needed for scaling and load balancing when consuming messages. All consumers in one group jointly read data from a topic, with each partition of the topic assigned to exactly one consumer from the group. This ensures parallel processing and guarantees that each message will be processed by exactly one consumer.

Broadcast (sending all messages to each consumer) in Kafka is not implemented directly, as Kafka is oriented towards a load-sharing model. To achieve broadcast, each consumer must be in its own unique Consumer Group. Then, each of them will receive all messages from the topic independently of others.

Example:

• A topic with 3 partitions
• 3 consumers in one group — each reads its own partition
• 3 consumers in 3 different groups — each receives all messages from all partitions (broadcast)

// Creating a consumer with a unique group for broadcast
Properties props = new Properties();
props.put("group.id", "unique-group-id");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("topic-name"));
// Each such consumer will receive all messages