What is a consumer group in Kafka?

A consumer group in Kafka is a group of consumers that jointly read data from a topic. Each partition of the topic is distributed among the consumers in the group so that each partition is read by only one consumer from the group. This allows for scaling message processing and load balancing. If there are multiple consumers in the group, they process different partitions in parallel, and if one consumer fails, its partitions are redistributed among the remaining consumers.

Example in Go using the sarama library:

consumerGroup, err := sarama.NewConsumerGroup(brokers, "my-group", config)
// Message processing logic is implemented in the handler