How did you monitor Kafka? Were there any incidents?

A comprehensive approach is usually used for Kafka monitoring, involving tools such as Prometheus, Grafana, Kafka Manager, Confluent Control Center, or built-in JMX metrics.

Key metrics for monitoring:

• Broker status (uptime, CPU load, memory).
• Producer and consumer latencies.
• Partition status (leaders, ISR — in-sync replicas).
• Consumer lag — critical for timely message processing.
• Errors and exceptions in logs.

Incidents are often related to:

• Loss of partition leader, causing data inaccessibility.
• Growth of consumer lag due to consumer issues.
• Broker overload.

To respond quickly to incidents, alerts are set up based on key metrics, for example, if lag exceeds a threshold or a broker stops responding.

Example of setting up an alert in Prometheus:

alert: HighConsumerLag
expr: kafka_consumergroup_lag > 1000
for: 5m
labels:
  severity: critical
annotations:
  summary: "High consumer lag"
  description: "Consumer lag for group {{ $labels.group }} exceeds 1000 messages"

This monitoring helps quickly identify and resolve issues in Kafka operation.