Abstract case: the service crashes (OOM kill), you see no errors, but the last status in the pod indicates a memory shortage. What is the sequence of actions for diagnostics and stabilization?

If the service crashes with an OOM kill error and there are no obvious errors in the logs, but the pod status indicates a lack of memory, the sequence of actions will be as follows:

1. Gather information about memory consumption:

   • Use kubectl describe pod <pod-name> to check events and status.
   • View memory metrics via kubectl top pod <pod-name> or through a monitoring system (Prometheus, Grafana).

2. Analyze resource limits and requests:

   • Check what memory limits are set in the pod manifest (requests and limits).
   • If the limit is too low, increase it.

3. Diagnose memory leaks:

   • For a Go service, collect a memory dump (heap dump) using pprof.
   • Analyze the dump for leaks or abnormal growth in consumption.

4. Code review:

   • Look for places where data might be accumulating in memory.
   • Optimize algorithms, use more efficient data structures.

5. Temporary stabilization measures:

   • Increase the memory limit in Kubernetes to prevent frequent crashes.
   • Implement restarts with backoff to recover the service.

6. Monitoring and alerts:

   • Set up memory usage monitoring and alerts for threshold breaches.

Example command to collect a memory profile in Go:

import (
    "net/http"
    _ "net/http/pprof"
)

func main() {
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // main service code
}

Then connect to http://localhost:6060/debug/pprof/heap to download the dump for analysis.