When a server "feels slow," the first question is which subsystem is actually hitting a ceiling: CPU, disk, memory, or network. Profiling is not guesswork — it is a systematic walk through these four points with specific commands and specific numbers in the output.
Basic Diagnostic Tools
| Tool | What it shows | Installation |
|---|---|---|
| top / htop | CPU, RAM, processes | htop (apt) |
| iostat | Disk I/O | sysstat (apt) |
| vmstat | Memory, swap, CPU | procps (built-in) |
| netstat / ss | Network connections | net-tools / iproute2 |
| iotop | Top processes by I/O | iotop (apt) |
| dstat | All-in-one | dstat (apt) |
Diagnosing High CPU Usage
# Find the process with the highest load
top -b -n 1 | head -20
# See what the process is doing
strace -p PID -c
# Flamegraph (if perf is installed)
perf record -g -p PID sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg
It is also worth checking load average against the core count: nproc shows how many there are, and if the first number from uptime stays noticeably above that count for hours rather than minutes, that is no longer a spike but a systemic CPU shortage, and profiling a single process will not substitute for an upgrade or code optimization.
Diagnosing a Slow Disk
# I/O statistics per device
iostat -x 1 5
# Which process is loading the disk
iotop -o -b -n 5
# Checking RAID / SMART status
smartctl -a /dev/sda
In iostat -x output, the key column is %util: a value close to 100% means the device is busy almost constantly, while the await column shows the average request latency in milliseconds — a rising await with unchanged util usually points to the disk itself degrading, not just growing load.
Diagnosing Insufficient Memory
# Memory summary
free -h
# Top processes by RAM
ps aux --sort=-%mem | head -10
# OOM killer log (who was killed)
dmesg | grep -i "killed process"
Common Messages and Metrics Explained
The line Out of memory: Killed process 4821 (mysqld) total-vm:2145728kB in dmesg output is not an error from mysqld — it is a kernel decision: the OOM killer picked the process using the most memory and forcibly terminated it to keep the whole system from freezing. After such an event, look at what caused the memory growth — a leak, a bad query, or simply not enough RAM for the current load — rather than just restarting the service.
High %wa (iowait) in top with low %us and %sy means the CPU is idle not because it has nothing to do, but because it is waiting on the disk to respond. A CPU upgrade will not help here — the bottleneck is physically in the disk subsystem, or in a specific process with excessive I/O, which is found through iotop.
Load average above the core count with normal CPU and I/O often means not computational load but a queue of processes waiting on an unavailable resource: a locked file, a semaphore, or a slow external call (for example to a database or an API). In that situation, ps aux will show processes in state D (uninterruptible sleep) — those are what drive the load average, not CPU usage.
Comparing Diagnostic Approaches
| Approach | Depth | System load | When to use |
|---|---|---|---|
| top / htop / free / df | Surface-level, a snapshot right now | Minimal | The first minute of diagnosis |
| iostat / vmstat in a loop | Medium, a trend over minutes | Low | Confirming a suspicion about disk or memory |
| strace / iotop filtered by PID | Deep, a specific process | Medium, may slow the process down | Finding the culprit among dozens of processes |
| perf + flamegraph | Maximum, the call stack | Noticeable under CPU-bound load | Optimizing specific code, not a one-off check |
When Profiling on the Server Itself Is Not Enough
- The problem shows up rarely and unpredictably — a single run of
topduring a quiet moment will show nothing; you need continuous metric collection like a monitoring system with history. - The bottleneck is not the VDS itself but an external dependency (slow DNS, a remote API, the network provider) — local tools will only show that the process is "waiting," without explaining what the network outside the server is waiting for.
- The load is spread across several servers — profiling a single VDS will not reveal the overall balancing picture or the load skew between nodes.
- You need the exact cause of application code degradation, not system resources — here system utilities fall short of application-level APM tools.
Verifying the Result
- After applying a fix (a database index, added swap, a changed config), repeat the same set of commands and compare the actual numbers: disk
%util,load average,%wa— the drop should be measurable, not just a feeling that things got faster. - Check that the problem did not just move somewhere else: reducing CPU load sometimes simply clears the way for a disk request queue that was previously hidden behind a slower CPU.
- Leave monitoring running for at least a day after the change — a one-off check right after the fix will not show behavior under an evening or nighttime peak.
- Record a baseline of the metrics in a calm state — without reference numbers for "normal" load, the next problem will have to be diagnosed from scratch.