Why are partitions needed in Kafka? How to choose their number? What is replication and what does it affect?

Partitions in Kafka are needed for scaling and performance improvement. A topic is divided into several partitions, which can be processed in parallel by different brokers and consumers. This allows increasing throughput and enabling parallel message processing.

The number of partitions is chosen based on expected load, the number of consumers, and latency requirements. The more partitions — the higher the parallelism, but also the higher the management overhead. Usually, the number of partitions is chosen to be a multiple of the number of consumers for optimal distribution.

Replication is copying partition data to multiple brokers to ensure fault tolerance. If one broker fails, data is available on other replicas. Replication affects reliability and availability but increases write latency and requires additional disk space.

Summary:

• Partitions — for parallelism and scalability
• Number of partitions — a balance between performance and complexity
• Replication — for reliability and fault tolerance

Example: a topic with 6 partitions and a replication factor of 3 means each partition is stored on 3 brokers.