How does Kafka work? Tell about delivery guarantees and idempotency.

Kafka works as a distributed event log where data is written to topics divided into partitions. Producers publish messages, and consumers read them, maintaining an offset.

Delivery guarantees:

• At most once — a message may be lost if not acknowledged.
• At least once — a message is guaranteed to be delivered but may be duplicated.
• Exactly once — when using idempotent producers and transactions, Kafka ensures exactly one delivery.

Idempotency: An idempotent producer in Kafka is one that, when resending the same message, does not create duplicates. This is achieved through unique message identifiers and sequence control.

Example of configuring an idempotent producer in Go using the sarama library:

config := sarama.NewConfig()
config.Producer.Idempotent = true
producer, err := sarama.NewSyncProducer(brokers, config)
if err != nil {
    panic(err)
}
// Now the producer guarantees exactly one delivery of messages

Thus, Kafka provides high reliability and scalability when working with data streams.