Can you explain what Apache Kafka is and how it is used for stream processing?

Apache Kafka is a distributed platform for processing real-time streaming data. It allows publishing, storing, and processing large volumes of messages with high throughput and low latency.

Main components of Kafka:

• Producer — sends messages to topics.
• Topic — a logical category or channel where messages are published.
• Broker — a Kafka server that stores and forwards messages.
• Consumer — subscribes to topics and receives messages.

Kafka is used for building systems where data arrives in a continuous stream, such as logging, monitoring, event processing, and microservices integration.

Example of usage in Python with the kafka-python library:

from kafka import KafkaProducer, KafkaConsumer

# Sending a message
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('my_topic', b'Hello Kafka')
producer.flush()

# Receiving messages
consumer = KafkaConsumer('my_topic', bootstrap_servers='localhost:9092')
for message in consumer:
    print(f'Received: {message.value.decode()}')

Thus, Kafka provides reliable and scalable processing of streaming data.