What NUMA is and why it matters on two processors

NUMA (Non-Uniform Memory Access) is the memory architecture of a two-socket server, where each CPU has "local" memory with fast access and "remote" memory on the other socket, reached through the inter-processor bus. The topic matters most for servers with two or more sockets — how the processor was chosen directly determines the number of NUMA nodes and their size.

If an application is not NUMA-aware, the OS scheduler can place a process on one socket while its memory sits on another. The result is a performance drop with no obvious cause in CPU monitoring.

How to check NUMA topology

lscpu | grep -i numa
numactl --hardware

The numactl --hardware command shows the number of nodes, the memory size on each, and the node distances. A value of 10 means access to the local node, 20-21 means access to a neighboring one: the higher the number, the more expensive a miss is.

The problem: memory on the "wrong" node

A typical scenario: a two-processor server with 256 GB of memory, an application started without any binding. The Linux scheduler moves the process between cores on different sockets while its allocated memory stays on the first node. Every access to the second socket goes through the inter-processor interconnect, and latency grows.

This is especially visible on database servers, where cache and buffers need to live next to the cores that read them thousands of times per second.

Binding processes: numactl and cpuset

Explicit binding of a process to a node solves the problem directly:

numactl --cpunodebind=0 --membind=0 /usr/bin/mysqld
numactl --interleave=all /usr/bin/myapp

The --cpunodebind flag pins the process to node 0's cores, --membind pins it to that same node's memory. The --interleave=all flag spreads memory evenly across all nodes — useful for applications that are not NUMA-aware and gain nothing from locality.

Parameter	What it does	When to use it
--cpunodebind	Pins the node's cores	Databases, JVM with a large heap
--membind	Pins the node's memory	Together with cpunodebind
--interleave=all	Interleaves memory across nodes	NUMA-unaware applications
--preferred	Prefers a node but does not forbid others	Flexible workloads

NUMA in databases and virtualization

PostgreSQL and MySQL recommend running one instance per NUMA node for large memory sizes, or explicitly binding the process with numactl if there is a single instance for the whole server. In virtualization (KVM, Proxmox) you should allocate a virtual machine's vCPUs and memory within a single physical node — otherwise the hypervisor runs into the same cross-node traffic problem.

Before changing any binding, measure the effect: run a test before and after with sysbench under a load close to the real one, not a synthetic single-threaded run.

NUMA tuning checklist

Check the topology with numactl --hardware before tuning, not after.
Bind the process to CPU and memory together, not separately.
For NUMA-unaware applications use --interleave=all instead of manual binding.
Measure latency and throughput before and after — the gain is not always obvious under light load.

numactl: Warning: cpu binding outside of valid numa node cpuset

libnuma: Warning: node argument 4 is out of range

mbind: Cannot allocate memory

numactl: command not found

Message	Cause	Fix
cpu binding outside of valid numa node cpuset	The specified cores don't belong to the node given in `--membind`	Check the node's core numbers with `numactl --hardware` before composing the command
node argument N is out of range	The command specifies a node number that doesn't physically exist on the server (for example, node 4 on a dual-socket machine with nodes 0 and 1)	Check the actual node count: `numactl --hardware \| grep available`
mbind: Cannot allocate memory	The requested NUMA node physically doesn't have enough free memory for a hard `--membind`	Use `--preferred` instead of `--membind` to allow allocation on another node when memory is short
numactl: command not found	The numactl package isn't installed — it's not part of the minimal install on most distributions	Install the package: `apt install numactl` or `dnf install numactl`

Ways to Manage NUMA Binding: A Comparison

Method	Scope	When to use it
numactl (for one process)	A single process being launched	Quickly testing a hypothesis and manually starting a database or service
cgroups cpuset	A process group, a systemd unit, a container	A persistent binding for a service that gets restarted by the system, not by hand
BIOS Node Interleaving	The entire system	Temporary diagnostics only — disables NUMA entirely, not for production

When NUMA Binding Has No Effect

The server is single-socket — the NUMA topology collapses to one node, and numactl --hardware will only show a node distance of 10; there's nothing to bind
The application's working set is smaller than one NUMA node's memory — the scheduler already keeps the process and its memory together almost all the time without manual intervention
The workload is heavily multithreaded and uses every core on the server evenly — hard binding to one node reduces parallelism here instead of speeding it up
The virtual machine already migrates between cluster nodes (live migration) — statically pinning vCPUs to a physical server's NUMA node loses its point once the VM moves to another machine

Verifying the Result

# Local and remote memory accesses by node
numastat -p $(pgrep -f mysqld)

# Memory access latency before and after binding
numactl --cpunodebind=0 --membind=0 -- perf stat -e task-clock,cycles ./benchmark

# Final memory distribution of the process across nodes
cat /proc/$(pgrep -f mysqld)/numa_maps | head -5

Run numastat -p PID before and after binding — the share of numa_foreign and other_node should drop noticeably
Measure the target load metric (requests per second, response time) both ways — without binding and with it — on the same traffic profile
Check via /proc/PID/numa_maps that most of the process's pages actually sit on the expected node rather than being spread evenly
Repeat the measurement after a day of real load — a synthetic test right after startup doesn't always show the same behavior as a warmed-up cache under steady state

Setting up virtualization itself, where a physical server's NUMA topology is passed through to guest machines, is covered in virtualization on a dedicated server.