Sobes.tech
Senior

What does 'not keeping state on the broker' mean in the context of Kafka?

sobes.tech AI

Answer from AI

In the context of Kafka, the phrase "not keeping state on the broker" means that Kafka brokers do not store any business logic state or consumer state information, apart from the message queue itself and topic metadata. All state related to message processing (such as consumer offsets, aggregated data, session states) is stored on the client side or in specialized storage systems (e.g., Kafka Streams State Stores or external databases).

This allows brokers to remain as simple and scalable as possible, while the state processing logic is transferred to applications that can flexibly manage and recover their state.

Example: a consumer's offset is stored in Kafka, but the application is responsible for how and when to commit it, and the business logic state (e.g., counting the number of events) is stored locally or in a separate store.