What exactly did you tune in Kafka? Which specific parameters did you configure?

When tuning Kafka, parameters that affect performance, reliability, and latency are usually configured. Here are some key parameters that are often adjusted:

• broker.config:

  • num.network.threads — number of threads for processing network requests.
  • num.io.threads — number of threads for input/output operations.
  • log.retention.hours and log.retention.bytes — log retention parameters to control volume and duration of storage.
  • log.segment.bytes — size of log segments, affecting write performance.

• producer.config:

  • acks — level of acknowledgment for writes (e.g., all for maximum reliability).
  • batch.size — batch size for sending messages.
  • linger.ms — delay before sending a batch to increase efficiency.
  • compression.type — message compression type (gzip, snappy, lz4).

• consumer.config:

  • fetch.min.bytes and fetch.max.wait.ms — parameters affecting latency and data volume during reads.
  • max.poll.records — maximum number of messages per poll call.

• Others:

  • Configuring replication and ISR for fault tolerance.
  • JVM parameters (if Kafka runs on JVM), such as heap size.

Example of producer configuration for increased throughput:

acks=all
batch.size=16384
linger.ms=5
compression.type=snappy

Depending on specific requirements (latency, reliability, throughput), parameters may vary.