How to find a memory leak in a production service if it cannot be reproduced locally?

To diagnose a memory leak in a production service when it cannot be reproduced locally, the following approaches can be used:

1. Collect memory profiles (heap dumps) in production. In Go, you can use the runtime/pprof package or built-in tools in the service (e.g., an HTTP endpoint for profiles). Take a heap profile when a leak is suspected.

2. Analyze profiles with pprof. Load the obtained profile into go tool pprof and analyze which objects occupy memory and are not released.

3. Monitor memory metrics. Use Prometheus or other monitoring systems to track memory consumption growth, GC pause times, and the number of goroutines.

4. Logging and tracing. Add logging for creation and deletion of key objects to understand where objects are not released.

5. Use dynamic analysis tools. For example, delve for debugging, or third-party APM systems with Go support.

6. Replicate production conditions. Try to reproduce the load and environment as closely as possible to production to catch the leak locally.

Example command to collect a heap profile in production:

import (
	"net/http"
	_ "net/http/pprof"
)

func main() {
	go func() {
		http.ListenAndServe(":6060", nil) // pprof is available at http://localhost:6060/debug/pprof/
	}()
	// rest of the code
}

Then, in production, you can get the profile:

curl http://localhost:6060/debug/pprof/heap > heap.prof

And analyze it locally:

go tool pprof heap.prof
(pprof) top