Server RAM testing: memtest86+ and EDAC counters

Server RAM testing: memtest86+ and EDAC

There are two ways to test memory in a dedicated server. Offline, by booting memtest86+ from an external image: it writes and reads patterns across every address and pinpoints the defect, but the server is down while it runs. Online, by reading the counters of the Linux kernel EDAC subsystem: it records every error ECC corrected and catches a degrading module weeks before it fails.

EDAC versus memtest86+: what each one catches

EDAC (Error Detection And Correction) is a set of kernel drivers for specific Intel and AMD memory controllers. They do not test memory; they read what the controller has already counted: how many single-bit errors were corrected (CE) and how many double-bit errors could not be (UE). Memory keeps running your real workload rather than a synthetic pattern.

memtest86+ works the other way round: it boots instead of the OS, takes all memory for itself and runs move inversions, random patterns and walking-ones tests over it. That is how you find defects ECC silently corrects, plus addressing faults where a write to one address corrupts another.

When you accept a new server, run both checks before migrating any data — the full order of work is covered in the article on accepting a dedicated server.

Online checks through EDAC

Counters in sysfs

Confirm the driver is loaded, then read the counters. On Xeon E3 and E5 platforms it is usually sb_edac, ie31200_edac or skx_edac.

# is the EDAC driver loaded?
lsmod | grep -i edac

# totals per memory controller
grep -H . /sys/devices/system/edac/mc/mc*/ce_count
grep -H . /sys/devices/system/edac/mc/mc*/ue_count

# per-slot breakdown: error count and the BIOS label of the slot
grep -H . /sys/devices/system/edac/mc/mc*/dimm*/dimm_ce_count
cat /sys/devices/system/edac/mc/mc0/dimm0/dimm_label

Zeros everywhere is the normal state. Treat any non-zero ue_count as an incident. A non-zero ce_count is not a verdict on its own; the trend matters. Record the value, wait a day, read it again.

memtester: checking without a reboot

memtester only tests the memory it manages to allocate, so the kernel and pages already in use stay untested. For a fast triage that is enough: if the defect falls inside the tested region, it surfaces within minutes.

# free memory in megabytes
free -m

# test 4 GB, 3 passes, write output to a file
memtester 4096M 3 > /var/log/memtester.log 2>&1

# how it ended
grep -i -E 'failure|ok' /var/log/memtester.log | tail -n 20

Running memtest86+ offline

A full test requires booting instead of the OS. On a dedicated server you do that through IPMI and KVM-over-IP: mount the ISO as a virtual drive and boot from it. KVM/IPMI access costs $5/month on every plan except Dedicated Enterprise US, where it is already included.

The other route is installing the package into the system, which adds an entry to the boot menu.

# Debian/Ubuntu: add a memtest86+ entry to the GRUB menu
apt-get install -y memtest86+
update-grub

# confirm the entry exists
grep -i memtest /boot/grub/grub.cfg

Plan for the runtime: a full pass over 16 GB of DDR4 takes roughly two to three hours, and 64 GB on Enterprise US takes close to a day. Budget at least two consecutive passes, or four when hunting an intermittent defect.

What to do about the errors you find

• One CE per month. Record it and watch. This can be a cosmic ray particle, which is exactly what ECC exists for.
• Dozens of CEs a day on one DIMM. The module is degrading; schedule a replacement in the next maintenance window.
• Any UE. Drain the workload now: an uncorrected error may already sit in a database file or a cache page.
• Errors move between slots. The modules are not the problem: look at the memory controller, the CPU and the timing settings in BIOS/UEFI.

Why ECC is worth it and how server memory differs from desktop memory is covered separately in the article on ECC memory in a server. Memory capacity per plan is listed on the dedicated servers page.

Key takeaways

• EDAC counters warn you early with no downtime; memtest86+ gives proof but needs the server stopped.
• Track the trend of ce_count per slot rather than the bare fact of a non-zero counter.
• Any UE justifies taking the server out of service without waiting for a maintenance window.
• For the offline test mount the ISO over IPMI and plan for at least two full passes.