How was Kafka deployed and what did the tuning involve?

The typical steps for deploying Apache Kafka are:

1. Infrastructure preparation: selecting servers or clusters, configuring network and high-performance disk subsystems.
2. Installing Kafka and Zookeeper: Kafka depends on Zookeeper for coordination, so a Zookeeper cluster is deployed first.
3. Broker configuration: setting parameters such as broker.id, log.dirs, num.partitions, replication.factor.
4. Security setup: TLS, authentication (SASL), authorization.

Kafka tuning includes:

• Performance parameter adjustments:
  • num.network.threads and num.io.threads for request processing optimization.
  • Partition sizes and counts for load balancing.
• Disk subsystem optimization: using SSDs, configuring filesystem parameters.
• Retention policy configuration: message retention time and log sizes.
• Monitoring and logging: setting up metrics for performance tracking and timely response.

An example of tuning in server.properties:

num.network.threads=3
num.io.threads=8
log.retention.hours=168
log.segment.bytes=1073741824

Ultimately, tuning depends on specific load requirements and infrastructure.