How did you ensure the absence of duplicate messages in Kafka?

To ensure the absence of duplicate messages in Kafka, I apply several approaches:

• Idempotent Producer: I enable idempotency on the Kafka producer side, which guarantees that resending the same message will not create a duplicate.
• Using Unique Message Keys: so that messages with the same key are processed sequentially and not duplicated.
• Consumer-side Processing: I implement idempotency logic during message processing, for example, by storing processed message identifiers in a database.
• Kafka Transactions: when necessary, I use transactional producers and consumers for atomic message processing.

Example of configuring an idempotent producer in Go using the sarama library:

config := sarama.NewConfig()
config.Producer.Idempotent = true
producer, err := sarama.NewSyncProducer(brokers, config)
if err != nil {
    panic(err)
}
// now the producer guarantees no duplicates on resend