How did you implement parallel request processing with Kafka? What was the implementation?

For parallel processing of requests using Kafka, multiple consumers are usually used, grouped into a consumer group. Each consumer in the group receives its own part of the topic partitions, allowing messages to be processed in parallel and scalable.

The implementation includes the following steps:

• Creating a topic with multiple partitions to ensure parallelism.
• Running multiple instances of consumers in one group so Kafka distributes partitions among them.
• Each consumer processes messages from its partitions independently.
• Using asynchronous processing and commit offset to guarantee processing.

Example in Java using KafkaConsumer:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "my-group");
props.put("enable.auto.commit", "false");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("my-topic"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    records.forEach(record -> {
        // Processing message
        System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
    });
    consumer.commitSync();
}

Thus, scaling is achieved by increasing the number of partitions and the number of consumers in the group.