How to configure producer and consumer interaction in Kafka if the consumer service is replicated on 3 instances and updates product prices? How to avoid race conditions?

If you have 3 instances of a service consumer updating product prices in Kafka, to avoid race conditions, you need to ensure that each message is processed by exactly one instance and updates do not conflict.

Main approaches:

1. Use a consumer group

   • All 3 instances are part of one consumer group.
   • Kafka guarantees that each partition of the topic will be processed by only one instance.
   • Thus, price messages for a single product (if the partition key is the product ID) will go to one partition and be processed sequentially by one instance.

2. Use a partition key

   • Make the product ID the message key.
   • This guarantees that all updates for one product will go to one partition and be processed sequentially.

3. Idempotency of updates

   • Updates should be idempotent so that reprocessing does not lead to errors.

4. Optionally: use external locking

   • If updates involve shared resources outside Kafka, distributed locks (e.g., based on Redis or Zookeeper) can be used.

Thus, proper configuration of the consumer group and partitioning by product key solves the race condition problem during price updates.