How does Kafka record messages and how does it transmit them?

Messages are sequentially recorded into segment files in the log. When each message is written, Kafka appends it to the end of the current active segment. Each segment has a fixed or configurable size, upon reaching which Kafka creates a new segment. The message offset is a unique identifier within a partition and represents its position in the log.

Message transfer is based on a pull mechanism: consumers independently request data from brokers. Each consumer group maintains its own offset for each partition, tracking how far it has read. Kafka does not store the state of each consumer individually, allowing the number of consumers to scale independently of the number of partitions or brokers. Brokers deliver messages from the requested offset to the last available.

Kafka uses the zero-copy principle for efficient data transfer from the filesystem to the network socket, bypassing user-space buffers, which significantly reduces the overhead of data copying.

// Example consumer request
// consumer.poll(Duration.ofMilliseconds(100))
// The consumer requests new messages starting from its current offset