Tell me about your experience with the observability stack Prometheus, Grafana, Loki. Provide an example of a non-standard alert.

Prometheus, Grafana, and Loki are a set of monitoring and logging tools I have used in several projects.

Prometheus is responsible for collecting metrics from applications and infrastructure. It allows setting up alerts based on these metrics. Grafana is used for visualizing data from Prometheus and other sources, creating user-friendly dashboards. Loki is a system for collecting and searching logs, integrated with Grafana, which enables linking metrics and logs.

An example of a non-standard alert:

In one project, I configured an alert that triggered not just when CPU usage exceeded a threshold, but when multiple conditions were met: if the average CPU load over 5 minutes was above 80% and the number of errors in logs (collected via Loki) exceeded a set level. This helped identify situations where load genuinely affected stability, rather than just peak CPU values.

An example of a Prometheus alert rule (simplified):

- alert: HighCpuAndErrorRate
  expr: avg_over_time(cpu_usage[5m]) > 0.8 and sum by (job) (rate({job="app"} |= "error"[5m])) > 10
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High CPU load and error increase"
    description: "CPU > 80% and error count > 10 over the last 5 minutes"

This approach helps reduce false alarms and respond more quickly to real issues.