Why is Kafka needed and how does it scale horizontally?

Kafka is a distributed data streaming platform designed for processing large volumes of messages with high throughput and low latency. It is used for building messaging systems, stream analytics, microservices integration, and more.

Horizontal scaling in Kafka is achieved through:

• Partitioning topics: each topic is divided into multiple partitions, which can be stored and processed on different brokers.
• Adding brokers: new servers (brokers) are added to the cluster, and partitions are redistributed among them.
• Partition replication: for fault tolerance, partitions are replicated across multiple brokers.

Thus, the load is distributed across multiple nodes, ensuring scalability and system resilience.

Example: if a topic has 10 partitions and there are 5 brokers in the cluster, the partitions are distributed among the brokers, and when new brokers are added, some partitions may be moved to balance the load.