Junior
What is Kafka?
sobes.tech AI
Answer from AI
A distributed, fault-tolerant publish-subscribe messaging system developed by LinkedIn and became an Apache project. It is used for building real-time data pipelines and streaming applications.
Main components and concepts:
- Broker: Kafka server. A Kafka cluster consists of one or more brokers.
- Topic: A category to which messages are published. Logically separates data streams.
- Partition: A topic is divided into partitions. Each partition is an ordered, immutable sequence of messages. Messages within a partition are assigned a unique offset.
- Producer: An application that publishes messages to a topic.
- Consumer: An application that subscribes to a topic and consumes messages.
- Consumer Group: A group of consumers that jointly consume messages from one or more topics. Messages from a partition are delivered to only one consumer in the group.
- Zookeeper: Used for coordinating the Kafka cluster (managing brokers, topics, partitions, consumer group offsets). In later versions, Kafka's role of Zookeeper is partially replaced by the built-in consensus service KRaft.
Kafka provides high throughput, scalability, and fault tolerance through partition replication across brokers. Messages are stored on disk and available for consumers to read for a specified period.
Use cases:
- Collecting and processing logs in real-time.
- Streaming data processing (via Kafka Streams, KSQL).
- Decoupling services (microservice architecture).
- Synchronizing data between systems.