How to localize a timeout problem in a distributed system? Where to look?

To localize the timeout issue in a distributed system, you need to sequentially check several levels:

1. Logs and metrics — examine the logs of the services involved in the chain, and metrics (latency, error rate). This will help understand at which stage the delay occurs.

2. Distributed tracing — if a tracing system (e.g., Jaeger, Zipkin) is used, look at where exactly the request 'hangs' or exceeds the timeout.

3. Network delays and errors — check the network status between services, possible packet losses, DNS issues, firewalls.

4. Resources and load — ensure that services are not overloaded with CPU, memory, or experiencing locks.

5. Timeout configurations — verify that timeouts on clients and servers are set correctly and are consistent.

6. External dependencies — if there are calls to external APIs or databases, check their availability and response times.

Thus, you should look into logs, metrics, traces, network settings, and timeout configurations to understand where exactly the delay occurs.