How would you troubleshoot a server under high load…

I will start with analyzing system metrics to identify the causes of high load.

Key indicators for analysis:

load average: the average number of processes waiting to be executed. A high value indicates CPU overload or I/O wait.
CPU utilization: the percentage of CPU time used. It helps to understand which part of the CPU is busy with user processes, system calls, or is idle.
Memory utilization: the use of RAM. High consumption can lead to active swapping, which sharply reduces performance.
Disk I/O: the intensity of read/write operations on disk drives. High I/O activity can be a bottleneck.
Network activity: incoming and outgoing traffic, number of network connections. High load may be caused by network issues or a DoS attack.
Process list: list of running processes, their resource consumption, and status. It helps to identify the specific process causing the load.

Diagnostic utilities:

top / htop: Interactive monitoring of system resources and processes in real-time. Allows sorting processes by CPU, memory, etc.
```
# top -c // display full command line
```
vmstat: Report on virtual memory, processes, CPU, and I/O activity.
```
# vmstat 1 // output every second
```

iostat: Monitoring disk device activity and performance.

# iostat -xz 1 // extended output for disk and CPU, every second

netstat / ss: Information on network connections, routes, network interface statistics.
```
# netstat -tunapl // TCP/UDP connections, LISTEN sockets, ports, PID
# ss -tunapl
```

sar: Collection, reporting, and analysis of system activity (CPU, memory, disk, network). Often used for historical performance analysis.

# sar -u 5 5 // CPU every 5 seconds, 5 times
# sar -d 5 5 // Disk every 5 seconds, 5 times
# sar -n DEV 5 5 // Network every 5 seconds, 5 times

iftop / nethogs: Interactive monitoring of network traffic by interface or process.

# iftop -i eth0 // monitor interface eth0
# nethogs eth0 // monitor traffic by process on eth0

dstat: Universal utility combining features of vmstat, iostat, netstat.
```
# dstat -c -m -d -n // CPU, memory, disk, network
```
strace / lsof: Tracing system calls (strace) and list of open files (lsof) for a specific process. Useful for deep application behavior analysis.
```
# strace -p <PID> // trace process by PID
# lsof -p <PID> // open files of process by PID
```

The procedure will be iterative: identify bottleneck based on metrics, use the appropriate utility for detailed analysis, analyze the root cause (e.g., a specific process), and take corrective actions (e.g., restart or optimize the application, increase resources). It is also important to check application logs and system logs (journalctl, /var/log/*).

How would you troubleshoot a server under high load? What metrics would you analyze and what utilities would you use?