Crisp answer: Work through layers in order — CPU, memory, I/O, network. The goal is to identify the bottleneck, not to check everything at once.
The 60-second triage:
uptime # Load average vs CPU count
free -m # Memory: used, free, buffer/cache, swap
df -h # Disk space (full disk causes writes to fail and apps to hang)
df -i # Inode exhaustion (often missed — small files fill inodes first)
top # CPU and memory by process
iostat -xz 1 3 # Disk I/O: %util (saturation), await (latency), r/s, w/s
ss -tnp # TCP connections — too many CLOSE_WAIT or TIME_WAIT?
dmesg | tail -20 # Recent kernel messages: OOM kills, hardware errors, disk errors
CPU:
uptime
# 12:00:00 up 10 days, load average: 8.12, 6.45, 4.32
# If load > nproc, CPU is saturated
nproc # Number of CPUs
mpstat -P ALL 1 # Per-CPU breakdown
top # Sort by CPU, identify process
Memory:
free -m
# total used free shared buff/cache available
# 32000 28000 500 1200 3500 3800
# "available" is what matters, not "free"
# available includes reclaimable buffer/cache
vmstat 1 # si/so columns: swap in/swap out — if non-zero, swapping
cat /proc/meminfo | grep -i swap # Swap usage
High swap usage means the system ran out of physical memory. Performance degrades drastically when paging to disk. Check for memory leaks:
ps aux --sort=-%mem | head # Top memory consumers
cat /proc/<pid>/smaps_rollup # Detailed memory map for a process
Disk I/O:
iostat -xz 1
# Device: util% r/s w/s await r_await w_await
# sda: 98% 50 500 45ms 5ms 50ms
# %util approaching 100% = disk saturated
# await > 20ms = high latency for random I/O (normal for spinning disk, bad for SSD)
iotop -o # Which processes are doing I/O right now
lsof +D /path # What files are open in a directory
Network:
ss -tnp # TCP connections with processes
ss -s # Summary: established, time-wait, etc.
sar -n DEV 1 # Network throughput per interface
iftop # Real-time bandwidth by connection (install if needed)
netstat -i # Interface stats: errors, drops
High TIME_WAIT count is usually normal (it's TCP being well-behaved after
close). High CLOSE_WAIT means your application isn't closing connections —
that's a bug.
Application-level:
If the OS looks healthy but the app is slow:
strace -p <pid> -e trace=network,file # What syscalls is it making?
lsof -p <pid> # Open files and connections
cat /proc/<pid>/wchan # What kernel function is it waiting on?
journalctl -u <service> -n 100 # Recent service logs
The systematic decision tree:
Load average high?
→ Yes: CPU or I/O bound?
→ %wa high: disk I/O → iostat
→ %wa low, %cpu high: find process → top/pidstat
→ No: application-level issue → logs, strace
Memory available low?
→ Check for leaks, check swap → vmstat
Disk full?
→ df -h, df -i → find and clean up
Network errors?
→ netstat -i, ss -s
What to say in the interview:
"I work through the layers in order: CPU with uptime and top, memory with free and vmstat watching for swap, I/O with iostat — %util and await are the key numbers, inodes with df -i which people often forget. Network with ss and sar. Once I've identified the layer, I zoom in — if it's CPU I find the process, if it's I/O I find what's doing the writes with iotop. If the OS looks fine, the issue is in the application — logs and strace."