Architecture of Apache Kafka: main components, how producers and consumers are connected? What is rebalancing? How to ensure sequential message reading?

Apache Kafka is a distributed data streaming platform built on the following core components:

• Producer — an application that publishes (sends) messages to Kafka topics.
• Consumer — an application that reads messages from topics.
• Topic — a logical category or channel where producers send messages.
• Partition — a topic is divided into multiple partitions for scalability and parallel processing.
• Broker — a Kafka server that stores data and handles requests from producers and consumers.
• Consumer Group — a set of consumers that jointly read data from a topic, distributing partitions among themselves.

The connection between producers and consumers occurs through topics and partitions. Producers send messages to a specific topic, and consumers within a group receive messages from the partitions of that topic. Each partition is assigned to only one consumer in the group, ensuring parallel processing without duplication.

Rebalance — the process of redistributing partitions among consumers within a group. This occurs when consumers join or leave the group, or when the number of partitions changes. Rebalancing guarantees that each partition is assigned to exactly one consumer.

To ensure sequential message reading:

• Guarantee that messages from one partition are read sequentially by one consumer.
• Use message keys when sending, so that all messages with the same key go to the same partition (partitioning by key).
• Process messages in the consumer synchronously or with order control.

An example in Go using the sarama library (simplified):

consumer, err := sarama.NewConsumerGroup(brokers, groupID, config)
// In the ConsumerGroupHandler, the ConsumeClaim method is implemented, where messages from partitions are read sequentially
func (h *consumerGroupHandler) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for msg := range claim.Messages() {
        fmt.Printf("Message topic:%s partition:%d offset:%d key:%s value:%s\n", msg.Topic, msg.Partition, msg.Offset, string(msg.Key), string(msg.Value))
        sess.MarkMessage(msg, "")
    }
    return nil
}

Thus, sequence is guaranteed within a partition, and keys help direct related messages to the same partition.