Tell about Kafka: topics, partitions, consumer groups, delivery guarantees.

Apache Kafka is a distributed data streaming platform that allows publishing, storing, and processing data streams in real-time.

• Topics — logical channels where producers send messages. A topic can be thought of as a category or a data stream name.

• Partitions — each topic is divided into multiple partitions for scalability and parallel processing. Each partition is an ordered, immutable log of messages.

• Consumer Groups — a group of consumers that jointly read data from a topic. Each partition in the topic is assigned to exactly one consumer in the group, ensuring parallel processing without duplication.

• Delivery Guarantees:

  • At most once — a message may be delivered 0 or 1 times (fast but with possible loss).
  • At least once — a message is delivered at least once, with possible duplicates.
  • Exactly once — a message is delivered exactly once, achieved through idempotent producers and transactions.

Example in Go using the sarama library to send a message:

package main

import (
    "github.com/Shopify/sarama"
    "log"
)

func main() {
    producer, err := sarama.NewSyncProducer([]string{"localhost:9092"}, nil)
    if err != nil {
        log.Fatal(err)
    }
    defer producer.Close()

    msg := &sarama.ProducerMessage{
        Topic: "example-topic",
        Value: sarama.StringEncoder("Hello Kafka"),
    }

    partition, offset, err := producer.SendMessage(msg)
    if err != nil {
        log.Fatal(err)
    }
    log.Printf("Message is stored in partition %d, offset %d\n", partition, offset)
}