How was the race condition problem solved when processing a single record concurrently by multiple consumers?

The race condition when multiple consumers process the same record in parallel is addressed using synchronization mechanisms and locks.

Main approaches:

• Locking mechanisms:

  • Using atomic database operations (e.g., SELECT ... FOR UPDATE) to ensure that only one consumer processes the record.
  • External distributed locks (e.g., via Redis, Zookeeper, or etcd).

• Identification and marking of records:

  • When processing begins, the record is marked with a "processing" flag indicating the consumer.
  • Other consumers skip such records.

• Message queues with delivery guarantees:

  • Using message brokers that ensure each message is processed by only one consumer.

Example with a database:

BEGIN;
SELECT * FROM tasks WHERE status = 'new' LIMIT 1 FOR UPDATE SKIP LOCKED;
-- if a record is found, update its status to 'processing'
UPDATE tasks SET status = 'processing' WHERE id = ?;
COMMIT;

This approach prevents multiple consumers from processing the same record simultaneously.