How initramfs Handles mdadm RAID Assembly During Ubuntu Server Boot: Debugging Superblock Issues


3 views

When your Ubuntu Server fails to assemble RAID arrays during boot with the dreaded "Gave up waiting for root device" error, the culprit often lies in how the initramfs environment interacts with mdadm and disk superblocks. Here's what's happening under the hood:

# Typical error you might see
Gave up waiting for root device.
ALERT! /dev/disk/by-uuid/[UUID] does not exist. Dropping to a shell!

The key issue occurs when superblock metadata exists at both the disk level (/dev/sda) and partition level (/dev/sda2), causing mdadm to assemble arrays incorrectly. This explains why your manual assembly works when specifying exact partitions:

# Correct manual assembly commands
mdadm --stop /dev/md1
mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm --assemble /dev/md1 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2

To verify if you're experiencing superblock conflicts, examine both disks and partitions:

# Check for conflicting superblocks
mdadm --examine /dev/sda
mdadm --examine /dev/sda1
mdadm --examine /dev/sda2

If /dev/sda and /dev/sda2 show identical RAID metadata, you've found the problem.

Here's how to properly clean and rebuild your RAID configuration:

# First, backup your current mdadm.conf
cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.bak

# Zero out problematic superblocks
mdadm --zero-superblock /dev/sda
mdadm --zero-superblock /dev/sdb
mdadm --zero-superblock /dev/sdc
mdadm --zero-superblock /dev/sdd

# Recreate arrays with proper partition alignment
mdadm --create /dev/md0 --level=1 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm --create /dev/md1 --level=5 --raid-devices=4 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2

# Update mdadm.conf
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

# Update initramfs
update-initramfs -u

To ensure your arrays assemble correctly during boot, verify these configuration points:

# Check initramfs modules
cat /usr/share/initramfs-tools/hooks/mdadm

# Verify udev rules
ls -l /etc/udev/rules.d/*md-raid*

# Confirm mdadm is included in initramfs
lsinitramfs /boot/initrd.img-$(uname -r) | grep mdadm

For complex cases, you can modify the initramfs scripts to include custom assembly commands:

# Create a custom initramfs script
echo 'mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1' > /etc/initramfs-tools/scripts/local-premount/mdadm_custom
echo 'mdadm --assemble /dev/md1 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2' >> /etc/initramfs-tools/scripts/local-premount/mdadm_custom
chmod +x /etc/initramfs-tools/scripts/local-premount/mdadm_custom
update-initramfs -u

Before rebooting, test your configuration:

# Simulate initramfs environment
mkinitramfs -o /tmp/test.initrd
lsinitramfs /tmp/test.initrd | grep mdadm

# Check array assembly
mdadm --detail --scan --verbose

When your Ubuntu Server drops into initramfs with the dreaded "Gave up waiting for root device" message, the culprit is often misassembled RAID arrays. The symptoms you described - where /dev/md1 assembles with whole disks (/dev/sdX) instead of partitions (/dev/sdX2) - point directly to superblock conflicts.

The key observation comes from your mdadm --examine outputs showing identical superblock information for both /dev/sda and /dev/sda2. This indicates stale superblocks at the disk level interfering with partition-level RAID assembly.

To verify:

mdadm --examine /dev/sda
mdadm --examine /dev/sda2

If both show RAID metadata (especially with the same UUID), you've confirmed the problem.

The correct approach is to selectively remove superblocks only from the disk level while preserving partition-level metadata:

for disk in /dev/sd[a-d]; do
    mdadm --zero-superblock $disk
done

This differs from what you attempted because it specifically targets the disk devices without affecting their partitions.

After cleaning superblocks, update your configuration:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf
update-initramfs -u

Initramfs assembles RAID arrays using these components:

  • /etc/mdadm/mdadm.conf - Primary configuration file
  • initramfs-tools scripts in /usr/share/initramfs-tools/scripts/
  • mdadm binary in the initramfs

To check how arrays will assemble during next boot:

mdadm --assemble --scan --verbose --config=/etc/mdadm/mdadm.conf

For persistent issues, add these kernel parameters to debug:

boot=debug break=premount

This drops you to a shell during initramfs execution where you can:

cat /scripts/local-premount/debug
mdadm --detail-platform
mdadm --examine --scan

Ensure your mdadm.conf contains ARRAY lines with proper device references:

ARRAY /dev/md0 metadata=1.2 UUID=xxxx devices=/dev/sda1,/dev/sdb1
ARRAY /dev/md1 metadata=1.2 UUID=yyyy devices=/dev/sda2,/dev/sdb2

The devices= parameter is crucial for correct assembly.