How many partitions should a Kafka topic have?

The number of partitions in a Kafka topic depends on several factors:

• Required throughput: The more partitions, the higher the parallelism and throughput.
• Number of consumers in the group: Usually, the number of partitions should not be less than the number of consumers to ensure efficient load distribution.
• Hardware resources: A greater number of partitions increases the load on the controller and brokers.

The optimal number of partitions is a compromise between performance and management complexity. For example, small systems may suffice with 3-10 partitions, while high-load systems may require hundreds or more.

Example of setting up a topic with 10 partitions using the kafka-topics command:

kafka-topics --create --topic my-topic --partitions 10 --replication-factor 3 --bootstrap-server localhost:9092