Software RAID 1 on NVMe with mdadm

Linux software RAID, the kernel's md driver managed with mdadm, is the standard way to mirror NVMe drives on a dedicated server. It is part of the mainline kernel, boots without special drivers, and costs a little CPU time instead of a RAID controller. This guide covers when to use RAID 1 or RAID 10, how to inspect and build arrays, how to monitor them, and how to replace a failed NVMe drive without losing the ability to boot.

Commands are for Ubuntu 24.04 and Debian 12/13 and run as root. On AlmaLinux and Rocky Linux, the configuration file is /etc/mdadm.conf and the initramfs is rebuilt with dracut -f.

RAID 1 or RAID 10

RAID 1 writes every block to both drives and can serve reads from either, which helps parallel reads; write speed matches a single drive. RAID 10 stripes across two mirrors, so four drives give twice the usable capacity of a two-drive mirror and better write throughput. Parity RAID (5 or 6) offers more capacity but adds write overhead and long rebuilds, so for databases and VMs on four NVMe drives, RAID 10 is the usual choice.

RAID 0 has no redundancy and only suits scratch data. No RAID level is a backup: it protects against a dead drive, not against rm -rf, corruption written by software, or a compromised host.

If you run Proxmox VE, use its ZFS mirror instead, because Proxmox does not support mdadm; see Proxmox VE on a dedicated server.

Check what the installer built

Installers, including Debian's and Ubuntu's, can create md arrays during setup. Before changing anything, find out what exists:

cat /proc/mdstat
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
mdadm --detail /dev/md0

A healthy mirror looks like this:

Personalities : [raid1]
md0 : active raid1 nvme1n1p2[1] nvme0n1p2[0]
      3749557248 blocks super 1.2 [2/2] [UU]
      bitmap: 1/28 pages [4KB], 65536KB chunk

[2/2] [UU] means two members, both up. [2/1] with [U_] or [_U] means the array is degraded, and a member marked (F) has failed. During a rebuild, a recovery = line shows progress and an estimated finish time.

Then check how the server boots. UEFI firmware cannot read an md array in the default metadata format, so the EFI System Partition (ESP) lives outside RAID. Installers often create an ESP on each drive but mount and fill only one, at /boot/efi:

lsblk -o NAME,SIZE,PARTTYPENAME,MOUNTPOINTS
efibootmgr -v

If only one drive has a populated ESP and a boot entry, the server will not boot when that drive dies. The section on keeping both drives bootable fixes that.

Create an array

On a new server, the cleanest path is to let the installer build RAID 1. To build an array yourself, for example a data array on the two spare drives of a four-drive server, use partitions of type Linux RAID rather than whole disks, and leave a little space at the end so a replacement drive with slightly fewer sectors still fits:

sgdisk --zap-all /dev/nvme2n1
sgdisk -n 1:0:-100M -t 1:FD00 /dev/nvme2n1
sgdisk --replicate=/dev/nvme3n1 /dev/nvme2n1
sgdisk --randomize-guids /dev/nvme3n1

mdadm --create /dev/md1 --level=1 --raid-devices=2 --bitmap=internal \
  /dev/nvme2n1p1 /dev/nvme3n1p1

For RAID 10, partition four drives the same way and use --level=10 --raid-devices=4 with four partitions. The write-intent bitmap lets the array resync only the regions that changed after a crash or a briefly missing member.

Create a filesystem, mount it by UUID, and record the array so it assembles under the same name at boot:

mkfs.ext4 /dev/md1
mkdir -p /srv/data
echo "UUID=$(blkid -s UUID -o value /dev/md1) /srv/data ext4 defaults,nofail 0 2" >> /etc/fstab
systemctl daemon-reload && mount /srv/data

mdadm --detail --scan | grep /dev/md1 >> /etc/mdadm/mdadm.conf
update-initramfs -u

The array is usable immediately while the initial sync runs in the background; follow it with watch -n 5 cat /proc/mdstat.

Monitor the array

A mirror that degrades silently is a single drive waiting to fail, so make sure something tells you.

mdadm --monitor

Debian and Ubuntu run mdadm --monitor as the mdmonitor service. It reports events such as Fail and DegradedArray to the MAILADDR set in /etc/mdadm/mdadm.conf, which needs a working mail transfer agent on the server. If the server cannot send mail, a PROGRAM line runs an executable of your choice instead. mdadm passes it the event name, the array and, when relevant, the member device, which is enough to post to a chat or paging webhook.

MAILADDR root
PROGRAM /usr/local/sbin/md-alert

Restart the service after editing the file, then confirm that an alert actually arrives. The second command sends a TestMessage event for every array:

systemctl restart mdmonitor && systemctl status mdmonitor
mdadm --monitor --scan --test --oneshot

Scheduled checks and metrics

Debian and Ubuntu also schedule a periodic consistency check of md arrays. You can start one by hand and read the result when it finishes:

echo check > /sys/block/md0/md/sync_action
cat /sys/block/md0/md/mismatch_cnt

If you use Prometheus, the node exporter's mdadm collector exposes node_md_disks{state="failed"}; alert on any value above zero. Watch the drives themselves too, because smartd and nvme smart-log often warn before a drive fails. The Linux security baseline covers both.

Replace a failed NVMe drive

When a drive fails, the array keeps running on the survivor. Replace it promptly: until the rebuild completes, one more failure loses the data.

1. Identify the drive by serial number

NVMe names such as nvme0 and nvme1 are assigned at boot and can change, so identify the failed drive by its serial number, never by name alone:

mdadm --detail /dev/md0             # which member is faulty or removed
nvme list                           # device, model and serial number
ls -l /dev/disk/by-id/ | grep nvme  # serial-based names for each device

2. Remove it from every array

If the drive has partitions in several arrays, for example root and swap, fail and remove each one:

mdadm --manage /dev/md0 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2

If the device has already disappeared, mdadm /dev/md0 --remove detached clears the stale entry.

3. Request the replacement

Open a support ticket with the serial numbers of the failed drive and of the healthy one, so there is no doubt about which drive to swap. FlexDedi keeps spare parts on site in every location. Plan a maintenance window in case the swap needs a power cycle.

4. Copy the partition table to the new drive

After the swap, confirm which device is new and empty with lsblk and nvme list. Then copy the layout from the healthy drive:

sgdisk --replicate=/dev/nvme1n1 /dev/nvme0n1   # new drive first, healthy drive last
sgdisk --randomize-guids /dev/nvme1n1

5. Add it back and rebuild

mdadm --manage /dev/md0 --add /dev/nvme1n1p2
watch -n 5 cat /proc/mdstat

Rebuild speed is throttled by dev.raid.speed_limit_min and dev.raid.speed_limit_max, typically 1,000 and 200,000 KB/s; check yours with sysctl dev.raid. At about 200 MB/s, rebuilding a 3.84 TB mirror takes roughly five hours (estimate: 3,840,000 MB ÷ 200 MB/s ≈ 5.3 hours). NVMe drives can go much faster, so raising the ceiling shortens the exposure window, at some cost to I/O for running workloads:

sysctl -w dev.raid.speed_limit_max=2000000

6. Make the new drive bootable

The new drive's ESP is empty. Recreate it as described in the next section, after removing any /etc/fstab entry that refers to the old ESP's UUID.

Keep both drives bootable

Mirroring the root filesystem is not enough: if the drive with the active ESP dies, the firmware must find a bootloader on the other one. On UEFI, format the second ESP, copy the files and register a boot entry. Check the folder name under /boot/efi/EFI/ first, for example debian or ubuntu:

mkfs.vfat -F 32 /dev/nvme1n1p1
mkdir -p /boot/efi2
echo "UUID=$(blkid -s UUID -o value /dev/nvme1n1p1) /boot/efi2 vfat umask=0077,nofail 0 1" >> /etc/fstab
systemctl daemon-reload && mount /boot/efi2
cp -r /boot/efi/. /boot/efi2/
efibootmgr --create --disk /dev/nvme1n1 --part 1 --label "debian disk 2" --loader '\EFI\debian\shimx64.efi'

GRUB and shim updates only touch the first ESP, so copy again after them. An APT hook does it automatically:

cat > /etc/apt/apt.conf.d/99-sync-esp <<'EOF'
DPkg::Post-Invoke { "if mountpoint -q /boot/efi2; then cp -r /boot/efi/. /boot/efi2/; fi"; };
EOF

On Ubuntu, dpkg-reconfigure grub-efi-amd64 can instead register several ESPs so GRUB installs to all of them. On legacy BIOS systems, dpkg-reconfigure grub-pc lets you select both drives. Test the result in a maintenance window by choosing the second drive in the firmware boot menu through the KVM console.

TRIM on md RAID

RAID 1 and RAID 10 pass discard (TRIM) requests through to the member drives, which helps NVMe keep write performance and wear in check. Confirm that discard reaches the array, then use the periodic fstrim timer instead of the discard mount option, which issues TRIM on every delete:

lsblk --discard /dev/md0   # non-zero DISC-GRAN and DISC-MAX mean TRIM works
systemctl enable --now fstrim.timer
fstrim -av

One side effect: if a drive does not return zeros for trimmed blocks, a later consistency check can report a non-zero mismatch_cnt for space that no filesystem uses. nvme id-ns -H /dev/nvme0n1 shows how a drive reads deallocated blocks. A small, stable mismatch count after TRIM is not by itself a sign of corruption; a count that keeps growing on a busy array deserves investigation.

Run it on FlexDedi

Every FlexDedi server has at least two NVMe drives for RAID 1, for example Core 16 ($299/mo) with 2 × 1.92 TB Gen4 or Pro 32 ($749/mo) with 2 × 3.84 TB Gen5. For RAID 10, Pro 64 ($1,349/mo) has 4 × 3.84 TB Gen5, 7.68 TB usable, and Max 96 ($1,899/mo) has 4 × 7.68 TB, 15.36 TB usable. If a server will not boot after a disk change, the KVM console and rescue mode let you repair it, and replacement drives come from spare parts kept on site.