What is a partition in Kafka? Why are they created?

A partition in Kafka is a logical part of a topic, representing an ordered, immutable log of messages.

Each topic is divided into multiple partitions for:

• Parallel processing: different consumers can process different partitions simultaneously.
• Scalability: increasing the number of partitions allows scaling of throughput.
• Resilience: data in partitions can be replicated across different brokers for fault tolerance.

Thus, partitions provide load balancing and system reliability.