How can you eliminate duplication in Kafka with At Least Once?

When using the "At Least Once" message delivery model in Kafka, message duplicates are possible. To eliminate them, the following approaches are usually applied:

1. Idempotent consumers — processing messages in such a way that reprocessing the same message does not lead to undesirable effects. For example, storing unique identifiers of processed messages and ignoring duplicates.

2. Using message keys and topic compaction — allows keeping only the latest state for each key, reducing the impact of duplicates.

3. Idempotent writing to final storage — for example, using UPSERT operations when writing to a database or checking for the existence of a record before inserting.

Thus, the main idea is to make message processing resilient to repetitions, rather than trying to completely eliminate duplication at the Kafka level.