Rescue mode: recovering a server that will not boot

Rescue mode boots the server into a minimal Linux system from the network or from an image, bypassing the disks entirely. Your data is left alone: you mount the root partition by hand, enter it with chroot and repair whatever is blocking a normal boot. It is the standard way to fix fstab, GRUB, initramfs or a lost root password without reinstalling.

Rescue formats nothing and touches no partitions: everything that happens, you do yourself and by hand.
A software array has to be assembled with mdadm --assemble --scan; it does not come up on its own.
LVM volumes appear only after vgchange -ay.
Half of all boot failures are a typo in /etc/fstab or a UUID that changed after a drive swap.
If rescue does not help, the cause is usually hardware: check SMART and the SEL instead of reinstalling.

When rescue solves the problem and when it does not

Symptom	Likely cause	What to do in rescue
Console hangs on the GRUB line	bootloader or partition table lost	chroot and reinstall GRUB on both disks
Kernel panic, no init found	broken initramfs, wrong root=	rebuild initramfs, fix the GRUB config
Boot stops in emergency mode	an error in /etc/fstab	check UUIDs, remove the stale line
Array reported as inactive	a drive dropped, damaged superblocks	assemble the array manually, start the resync
Disk not detected at all	drive or backplane failure	rescue will not help; hardware diagnostics needed

How to get into rescue

There are two routes. The first is a network boot of a minimal system started by the provider on request: the server boots a PXE image and accepts SSH with a temporary password. The second is doing it yourself from the console by attaching a live ISO as a virtual drive, as described in the article on IPMI and KVM-over-IP. The KVM/IPMI option on ZevsHost costs $5 per month and is included in the Dedicated Enterprise US plan.

Find and mount the root partition

Do not mount the first partition you see. Start by looking at what is actually on the disks:

# which disks, partitions and filesystems the system sees
lsblk -f
blkid

# state of the software arrays
cat /proc/mdstat
mdadm --assemble --scan
mdadm --detail /dev/md0

# LVM volumes are inactive by default
pvs && vgs && lvs
vgchange -ay

Then mount the root and the service filesystems. Order matters: the root first, then everything that lives inside it.

mount /dev/md1 /mnt
mount /dev/md0 /mnt/boot
# on UEFI do not forget the ESP
mount /dev/nvme0n1p1 /mnt/boot/efi

for d in dev proc sys run; do mount --rbind /$d /mnt/$d; done
chroot /mnt /bin/bash

If chroot complains about a missing interpreter, you mounted the wrong partition: check with ls /mnt that the root holds etc, usr and var.

Common repairs

A broken fstab

# list every fstab entry that does not resolve
findmnt --verify --verbose

# compare the UUIDs in fstab with the real ones
blkid | sort > /tmp/real-uuid.txt
grep -v '^#' /etc/fstab | grep UUID

After a drive swap the UUID changes and the stale line stops the boot. Write in the new UUID or add the nofail option so a missing volume no longer blocks startup. The swap procedure itself is covered in the article on replacing a disk in a RAID array.

Bootloader and initramfs

# Debian/Ubuntu, inside the chroot
grub-install /dev/sda && grub-install /dev/sdb
update-grub
update-initramfs -u -k all

# RHEL, AlmaLinux, Rocky
grub2-install /dev/sda && grub2-install /dev/sdb
grub2-mkconfig -o /boot/grub2/grub.cfg
dracut -f --regenerate-all

Install the bootloader on both disks of the mirror: a machine whose GRUB lives only on the first drive will not come up after that drive fails.

Root password and logs

# reset the password inside the chroot
passwd root
passwd -S root

# errors from the previous boot, outside the chroot
journalctl --directory=/mnt/var/log/journal -b -1 -p err

Never re-create an array instead of assembling it. Running mdadm --create on existing disks rewrites the superblocks and can destroy the data; recovery needs mdadm --assemble. Any mkfs is equally dangerous: device names in rescue often differ from the familiar ones. Run lsblk -f before every writing command. Before leaving, check yourself: findmnt --verify returns no errors, cat /proc/mdstat shows [UU], and ls /mnt/boot contains the kernel and initrd of the right version. Restoring from a backup is cheaper than recovering from a wrong command.

Before you leave rescue

Exit the chroot and unmount everything in reverse order: umount -R /mnt.
Stop the array cleanly if you changed its membership: mdadm --stop /dev/md0.
Clear the network boot mode, or the server will start into rescue again.
After the first successful boot, check the drives using the method in the article on SMART disk diagnostics.

If the boot failure keeps coming back, the cause is almost always hardware. At that point it is worth requesting a drive replacement or a move to another configuration from the dedicated servers line-up.

Key takeaways

Rescue boots a separate system into memory and leaves your disks untouched.
Arrays are assembled with mdadm --assemble --scan; LVM is activated with vgchange -ay.
Most failures are fixed inside a chroot: fstab, GRUB, initramfs, the root password.
The bootloader always goes on both mirror disks.
mdadm --create and mkfs in rescue are the commands after which you restore from backup.