How do you troubleshoot high CPU on a Linux server

Crisp answer: Identify which process, then why — is it doing useful work (busy), spinning on a lock, waiting on I/O, or stuck in the kernel?

Step 1 — Identify the culprit

top                          # Interactive, press P to sort by CPU
htop                         # Better top, press F6 for sort options
ps aux --sort=-%cpu | head   # Static snapshot, sorted by CPU desc
pidstat 1 5                  # Per-process CPU every 1s for 5 intervals

Key columns in top:

%CPU — CPU usage (can exceed 100% on multi-core, means it's using multiple cores fully)
%us — user space, %sy — kernel/system, %wa — I/O wait, %st — stolen (VM context)

High %wa means the server is actually I/O bound, not CPU bound — the CPU is idle waiting for disk/network. Don't be fooled.

Step 2 — Is it one core or all?

mpstat -P ALL 1              # Per-CPU stats every 1s

If one CPU is at 100% and others are idle, the process is single-threaded or pinned to a core. If all CPUs are high, it's a genuinely CPU-intensive workload.

Step 3 — What is the process doing?

strace -p <pid> -c           # Summarise syscalls — is it doing useful I/O or spinning?
strace -p <pid>              # Live syscall stream (use sparingly — slows the process)
cat /proc/<pid>/status       # Process state, memory, threads
cat /proc/<pid>/wchan        # What kernel function the process is waiting on
ls -l /proc/<pid>/fd         # What files are open

If strace shows the process making syscalls, it's doing real work. If strace shows nothing or tight loops of a single syscall (like futex), it's often a deadlock or busy-wait.

Step 4 — Application-level profiling

Java: jstack <pid> for thread dump, JVM flight recorder
Python: py-spy top --pid <pid> (install separately)
Go: go tool pprof http://localhost:6060/debug/pprof/profile
Node.js: --inspect flag + Chrome DevTools

Step 5 — Is it a runaway process or expected load?

uptime                       # Load averages: 1m 5m 15m
# Load average > number of CPU cores = saturated
nproc                        # Number of CPU cores

If load average is trending down, the spike has passed. If it's trending up, something is growing unbounded.

Common causes:

Symptom	Likely cause
One process at 100%, tight loop	Infinite loop / runaway job
Many processes at 10-20%	Web server under load, normal
`%sy` (system) very high	I/O-heavy workload, kernel doing syscalls
`%wa` high	I/O wait — disk or network bound, not truly CPU
`%st` high	VM is being throttled by the hypervisor — cloud burst limit
Load average rising but CPU looks low	Too many processes, context switching overhead

What to say in the interview:

"Start with top or pidstat to identify which process. Check mpstat to see if it's one core or all. Then strace -p to see what syscalls it's making — is it doing real work or spinning? For application-level insight, language-specific profilers. And check load average vs number of cores to understand whether the system is saturated."

My notes