Tell me about the monitoring setup: metric collectors, alerts

System monitoring typically consists of three key components: metric collectors, storage and visualization system, and alerts.

Metric collectors are agents or libraries that gather data about the application's or infrastructure's state. In Go, the Prometheus client library is often used to export metrics. Metrics can include CPU load, memory usage, request count, response time, etc.

Alerts are rules that trigger when certain metric thresholds are reached. For example, if response time exceeds 500 ms or error percentage rises above 5%. Alerts are usually configured in the monitoring system (e.g., Prometheus Alertmanager) and can send notifications via Slack, email, or other channels.

An example of simple metric export in Go with Prometheus:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var (
    httpRequests = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"path"},
    )
)

func init() {
    prometheus.MustRegister(httpRequests)
}

func handler(w http.ResponseWriter, r *http.Request) {
    httpRequests.WithLabelValues(r.URL.Path).Inc()
    w.Write([]byte("Hello, world!"))
}

func main() {
    http.HandleFunc("/", handler)
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

Thus, metric collectors gather data, the monitoring system stores and visualizes it, and alerts notify about issues.