Why is a partition key needed? How does Kafka determine the partition by key?

The partition key in Kafka is used to determine which partition of a topic a specific message will go to. This is important for maintaining the order of messages with the same key and load balancing across partitions.

Kafka uses the key to compute a hash value, typically through a hashing function (e.g., murmur2). Then, this value is taken modulo the number of partitions to determine the partition index:

int partition = (key == null) ?
    randomPartition() :
    Math.abs(hash(key)) % numPartitions;

Thus, all messages with the same key will be guaranteed to go to the same partition, which is important for orderliness and processing consistency.