What methods of dealing with duplicates in Kafka exist?

In Kafka, message duplication can occur due to re-sending or re-consuming. The main ways to combat duplicates:

1. Idempotent producer — starting from Kafka 0.11, the producer can be configured as idempotent (enable.idempotence=true). This guarantees that resending the same message will not create duplicates in the topic.

2. Using unique keys and idempotency logic on the consumer side — if the producer is not idempotent, the consumer can filter duplicates by a unique message identifier.

3. Kafka transactions — allow atomic sending of multiple messages and committing them, helping to avoid partial duplicates.

4. Manual offset handling — the consumer can control offset commits only after successful message processing to prevent re-reading.

5. Using external systems for state storage — for example, databases with unique keys for filtering duplicates.

Thus, combating duplicates is a combination of producer settings, consumer configurations, and architectural solutions.