Why does Apache Kafka have high performance?

Kafka achieves high performance through several key mechanisms:

1. Sequential disk writing: Kafka brokers write data to logs (segment files) sequentially. Sequential writing is much faster than random, especially on traditional disks (HDD).

2. No intermediate buffering: Kafka does not buffer messages in broker memory before writing to disk. Messages are written immediately upon receipt, reducing latency and memory consumption.

3. Using OS Page Cache: Brokers actively utilize the operating system's page cache for reading data. This avoids repeated disk reads and significantly speeds up message delivery to subscribers. Data written to disk by the broker is automatically placed into the Page Cache.

4. Zero-copy: When sending data to subscribers, Kafka uses the "zero-copy" technique (implemented via sendfile in Linux). This avoids intermediate copying of data between kernel buffers and user space, transferring data directly from the broker's Page Cache to the consumer's network socket.

   // Example pseudocode illustrating the sendfile concept
   // (Not actual Kafka code, just demonstrating the idea)
   FileChannel fileChannel = ...; // File channel for reading from Kafka log
   SocketChannel socketChannel = ...; // Channel for sending data over the network
   
   fileChannel.transferTo(0, fileChannel.size(), socketChannel); // Zero-copy operation
   

5. Batching: Kafka allows sending and receiving messages in batches. This reduces overhead on network requests and disk operations. Producers can group multiple messages before sending, and consumers can read multiple messages in a single request.

6. Partitioning: Topics are divided into partitions. Each partition is processed independently, allowing load distribution among multiple brokers and parallel processing of read/write operations. This provides horizontal scalability.

7. Simple message format: Messages have a relatively simple structure, minimizing serialization and deserialization costs.

Together, these mechanisms enable Kafka to handle very large volumes of messages with low latency.