How to determine the proper functioning of microservices?

Monitoring and logging. Health metrics, errors, performance. Distributed tracing. Synthetic tests and readiness/liveness probes.

• Monitoring: Collection of service operation metrics.
  • HTTP response codes (2xx, 4xx, 5xx).
  • Response time latency.
  • QPS (Queries Per Second).
  • Resource usage: CPU, memory, network, disk.
  • Application-specific metrics (e.g., number of processed requests, queue depth).
• Logging: Centralized log collection. Log analysis for errors, warnings, critical events. Use of structured logging.
• Readiness Probes: Determine if the service is ready to accept traffic. If the check fails, the orchestrator (e.g., Kubernetes) stops routing requests to it.
• Liveness Probes: Determine if the service process is alive. If the check fails, the orchestrator restarts the pod.
• Distributed Tracing: Tracking the request path across multiple microservices. Helps identify bottlenecks and errors in the call chain. Tools: Jaeger, Zipkin, OpenTelemetry.
• Synthetic Testing: Simulating user scenarios externally by sending requests to services and checking responses.
• Circuit Breakers: Pattern that prevents cascading failures. If a dependent service is unavailable, the breaker opens, and services stop sending requests to it, allowing recovery time.
• Service Dependency Graphs: Visualization of relationships between microservices. Helps understand which services influence each other during failures.

Examples of metrics and queries (Prometheus Query Language):

# Average response time for requests to "my-service"
rate(http_request_duration_seconds_sum{service="my-service", status!="5xx"}[5m]) / rate(http_request_duration_seconds_count{service="my-service", status!="5xx"}[5m])

# Number of 5xx errors in the last minute
sum(rate(http_requests_total{service="my-service", status="5xx"}[1m]))

# CPU usage of a pod
sum(rate(container_cpu_usage_seconds_total{namespace="my-namespace", pod="my-pod-..."}[5m])) by (pod)