What tools have you used for service observability?

For observability of services in Go, I usually use a combination of several tools:

• Prometheus for metrics collection: easily integrates with Go through a client library, allows collecting custom metrics and building dashboards.
• Grafana for visualizing metrics from Prometheus.
• Jaeger or OpenTelemetry for distributed tracing, which helps understand delays and call chains between services.
• Logrus or Zap for structured logging with subsequent aggregation into ELK stack (Elasticsearch, Logstash, Kibana) or Loki.

Example of integrating Prometheus in Go:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var requestCount = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total number of HTTP requests",
    },
    []string{"path"},
)

func init() {
    prometheus.MustRegister(requestCount)
}

func handler(w http.ResponseWriter, r *http.Request) {
    requestCount.WithLabelValues(r.URL.Path).Inc()
    w.Write([]byte("Hello, world!"))
}

func main() {
    http.HandleFunc("/", handler)
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8080", nil)
}

This stack provides a complete view of the state and performance of services.