Skip to main content

Linux: server performance profiling

VDS / VPS · 13.03.2025 · 5 min read
Illustration for “Linux: server performance profiling”

When a server "feels slow," the first question is which subsystem is actually hitting a ceiling: CPU, disk, memory, or network. Profiling is not guesswork — it is a systematic walk through these four points with specific commands and specific numbers in the output.

Basic Diagnostic Tools

ToolWhat it showsInstallation
top / htopCPU, RAM, processeshtop (apt)
iostatDisk I/Osysstat (apt)
vmstatMemory, swap, CPUprocps (built-in)
netstat / ssNetwork connectionsnet-tools / iproute2
iotopTop processes by I/Oiotop (apt)
dstatAll-in-onedstat (apt)

Diagnosing High CPU Usage

# Find the process with the highest load
top -b -n 1 | head -20

# See what the process is doing
strace -p PID -c

# Flamegraph (if perf is installed)
perf record -g -p PID sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg

It is also worth checking load average against the core count: nproc shows how many there are, and if the first number from uptime stays noticeably above that count for hours rather than minutes, that is no longer a spike but a systemic CPU shortage, and profiling a single process will not substitute for an upgrade or code optimization.

Diagnosing a Slow Disk

# I/O statistics per device
iostat -x 1 5

# Which process is loading the disk
iotop -o -b -n 5

# Checking RAID / SMART status
smartctl -a /dev/sda

In iostat -x output, the key column is %util: a value close to 100% means the device is busy almost constantly, while the await column shows the average request latency in milliseconds — a rising await with unchanged util usually points to the disk itself degrading, not just growing load.

Diagnosing Insufficient Memory

# Memory summary
free -h

# Top processes by RAM
ps aux --sort=-%mem | head -10

# OOM killer log (who was killed)
dmesg | grep -i "killed process"
💡 Install htop + iotop + nethogs — these three tools are enough to diagnose 90% of real-time performance issues.

Common Messages and Metrics Explained

The line Out of memory: Killed process 4821 (mysqld) total-vm:2145728kB in dmesg output is not an error from mysqld — it is a kernel decision: the OOM killer picked the process using the most memory and forcibly terminated it to keep the whole system from freezing. After such an event, look at what caused the memory growth — a leak, a bad query, or simply not enough RAM for the current load — rather than just restarting the service.

High %wa (iowait) in top with low %us and %sy means the CPU is idle not because it has nothing to do, but because it is waiting on the disk to respond. A CPU upgrade will not help here — the bottleneck is physically in the disk subsystem, or in a specific process with excessive I/O, which is found through iotop.

Load average above the core count with normal CPU and I/O often means not computational load but a queue of processes waiting on an unavailable resource: a locked file, a semaphore, or a slow external call (for example to a database or an API). In that situation, ps aux will show processes in state D (uninterruptible sleep) — those are what drive the load average, not CPU usage.

Comparing Diagnostic Approaches

ApproachDepthSystem loadWhen to use
top / htop / free / dfSurface-level, a snapshot right nowMinimalThe first minute of diagnosis
iostat / vmstat in a loopMedium, a trend over minutesLowConfirming a suspicion about disk or memory
strace / iotop filtered by PIDDeep, a specific processMedium, may slow the process downFinding the culprit among dozens of processes
perf + flamegraphMaximum, the call stackNoticeable under CPU-bound loadOptimizing specific code, not a one-off check

When Profiling on the Server Itself Is Not Enough

  • The problem shows up rarely and unpredictably — a single run of top during a quiet moment will show nothing; you need continuous metric collection like a monitoring system with history.
  • The bottleneck is not the VDS itself but an external dependency (slow DNS, a remote API, the network provider) — local tools will only show that the process is "waiting," without explaining what the network outside the server is waiting for.
  • The load is spread across several servers — profiling a single VDS will not reveal the overall balancing picture or the load skew between nodes.
  • You need the exact cause of application code degradation, not system resources — here system utilities fall short of application-level APM tools.

Verifying the Result

  1. After applying a fix (a database index, added swap, a changed config), repeat the same set of commands and compare the actual numbers: disk %util, load average, %wa — the drop should be measurable, not just a feeling that things got faster.
  2. Check that the problem did not just move somewhere else: reducing CPU load sometimes simply clears the way for a disk request queue that was previously hidden behind a slower CPU.
  3. Leave monitoring running for at least a day after the change — a one-off check right after the fix will not show behavior under an evening or nighttime peak.
  4. Record a baseline of the metrics in a calm state — without reference numbers for "normal" load, the next problem will have to be diagnosed from scratch.
Was this article helpful?
← Back to Knowledge Base Ask Support