How to configure Kafka for 100 application instances with 10 threads each so that no one blocks each other?

For 100 application instances with 10 threads each to work efficiently with Kafka and not block each other, you need to properly design topics and partitions.

1. Number of partitions: Ensure that the number of partitions in the topic is not less than the total number of threads that will read data (100 instances × 10 threads = 1000 threads). This allows each thread to read from a separate partition and avoid blocking.

2. Consumer groups: Use different consumer groups for different tasks if you need to process the same messages in parallel without conflicts.

3. Load balancing: Kafka automatically distributes partitions among consumers in a group. When the number of threads or instances increases, partitions will be redistributed.

4. Idempotency and processing: Ensure idempotent message processing to avoid issues with re-delivery.

Example of configuring a consumer group in Go using the sarama library:

consumerGroup, err := sarama.NewConsumerGroup(brokers, "my-group", config)
// Launch 10 threads for each instance
for i := 0; i < 10; i++ {
    go func() {
        for {
            err := consumerGroup.Consume(ctx, []string{"my-topic"}, handler)
            if err != nil {
                log.Println("Error consuming:", err)
            }
        }
    }()
}

The main point is to ensure a sufficient number of partitions and to correctly use consumer groups so that each thread receives unique partitions and does not block others.