How to ensure message delivery exactly once, avoiding duplication?

To ensure exactly-once delivery of messages, it is necessary to combine several approaches, as this is a non-trivial task in distributed systems:

• Idempotency of message handlers: processing the same message multiple times should not change the final result.
• Unique message identifiers: each message has a unique ID that is stored in the database upon processing.
• Processing state storage: upon receiving a message, it is checked whether it has already been processed (by ID). If yes, it is ignored.
• Transactional acknowledgment: recording the processing result and marking the message as processed should be atomic.

Example in Python using a database to track processed messages:

import sqlite3

conn = sqlite3.connect('messages.db')
conn.execute('''CREATE TABLE IF NOT EXISTS processed_messages (msg_id TEXT PRIMARY KEY)''')

def process_message(msg_id, payload):
    cursor = conn.cursor()
    cursor.execute('SELECT 1 FROM processed_messages WHERE msg_id = ?', (msg_id,))
    if cursor.fetchone():
        print('Message already processed, skipping')
        return

    # Process the message
    print(f'Processing message: {payload}')

    # Save the processing fact
    cursor.execute('INSERT INTO processed_messages (msg_id) VALUES (?)', (msg_id,))
    conn.commit()

# Example call
process_message('12345', 'Data')
process_message('12345', 'Data')  # will be skipped

Thus, even with repeated delivery, the message will only be processed once.