How did the Kafka cluster learn about its configurations and pass them to clients? How is metadata composed?

The Kafka cluster learns about its configurations and transmits them to clients through a metadata mechanism, which is formed based on information about brokers, topics, and partitions.

Each broker, upon startup, reads its configuration files and registers in ZooKeeper (in older Kafka versions) or in its own internal coordination service (KRaft in newer versions). ZooKeeper stores information about the cluster's state, including the list of brokers, topics, partitions, and their leaders.

Metadata is structured information that brokers pass to clients in response to requests. It includes:

• List of brokers and their addresses
• Topics and number of partitions
• Which partition is led by which broker

Clients request metadata from any broker to know where to send messages or from where to read. Clients periodically update the metadata to account for changes in the cluster (e.g., adding brokers or changing partition leaders).

Example of a metadata request in Kafka Producer (Java):

Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
// When initializing, the producer requests metadata from brokers

Thus, metadata is formed based on the cluster state stored in ZooKeeper or KRaft and is transmitted to clients for proper operation with topics and partitions.