Who configured alerts and how does a service degradation turn into an alert?

Alerts are usually configured through monitoring and notification systems such as Prometheus with Alertmanager, Zabbix, Grafana, or Datadog. Service degradation refers to the deterioration of key metrics (for example, increased response time, error growth, decreased availability).

The process of turning a degradation into an alert looks like this:

1. Metrics collection — monitoring gathers data about the service's state.
2. Rule definition — threshold values are set, exceeding which is considered incorrect service operation.
3. State assessment — the monitoring system checks whether thresholds are exceeded.
4. Alert generation — if the condition triggers, an alert is created.
5. Notification — the alert is sent to responsible parties via email, messengers, SMS, or incident management system integrations.

Example of a rule in Prometheus Alertmanager:

- alert: HighErrorRate
  expr: job:request_errors:rate5m > 0.05
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "High error rate on the service"
    description: "Error exceeds 5% for 5 minutes"