How are consumer groups configured in Kafka? How many topics do you read from?

In our projects, consumer groups in Kafka are configured to ensure scalable and fault-tolerant message consumption. Usually, each service has its own consumer group, allowing multiple instances of the service to read from a single topic in parallel, distributing partitions among themselves.

The number of topics depends on the business logic, typically ranging from 1 to 5 topics, each responsible for a different type of event. The sources of topics are producers from various microservices or external systems.

An example configuration in Go using the sarama library:

config := sarama.NewConfig()
config.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategyRange

consumerGroup, err := sarama.NewConsumerGroup(brokers, "my-consumer-group", config)
if err != nil {
    log.Fatal(err)
}

topics := []string{"topic1", "topic2"}

// Start consumption in a goroutine
ctx := context.Background()
go func() {
    for {
        if err := consumerGroup.Consume(ctx, topics, handler); err != nil {
            log.Println("Error from consumer:", err)
        }
    }
}()

Thus, the consumer group ensures load balancing and guarantees that each message will be processed by exactly one consumer in the group.