Using ScyllaDB before the producer is an anti-pattern. How to organize deduplication correctly?

Using ScyllaDB directly before the producer for deduplication is an anti-pattern, as the database is not designed for fast and efficient filtering at the producer stage, which can lead to unnecessary load and delays.

The correct approach to deduplication:

1. Deduplication at the producer or intermediate layer — implement message filtering logic before writing to the database, using, for example, a cache with TTL (like Redis) to store unique message keys.

2. Use of idempotent keys — when sending messages to the queue or database, use unique identifiers so downstream systems can ignore duplicates.

3. Asynchronous deduplication — if real-time deduplication is not critical, you can implement a process to clean up duplicates on the consumer side or in a separate service.

Thus, deduplication should be moved out of the database and implemented at the producer or intermediate layer using specialized tools and algorithms, which will improve system performance and reliability.