How to Migrate from Linux Software RAID5 to RAID6 Without Data Loss: A Step-by-Step Guide for Sysadmins


3 views

Many sysadmins face the challenge of expanding their storage arrays while maintaining data availability. In this case, we have:

  • Original setup: 3x1.5TB disks in RAID5 (mdadm)
  • New hardware: 3 additional 1.5TB disks (same model recommended)
  • Goal: Convert to RAID6 without restoring from backup

Before beginning the conversion:


# Verify current array status
cat /proc/mdstat
mdadm --detail /dev/mdX

# Check filesystem integrity (if using ext4)
e2fsck -f /dev/mdXp1

# Ensure backup exists (even though we won't use it)
rsync -aHAXv /mnt/array/ /backup/location/

1. Adding New Disks to the Array


# Identify new disks (usually /dev/sdX)
lsblk

# Add disks one by one to avoid overheating
mdadm --add /dev/md0 /dev/sdd
mdadm --add /dev/md0 /dev/sde
mdadm --add /dev/md0 /dev/sdf

# Verify growth operation started
cat /proc/mdstat

2. Reshaping to RAID6 Layout


# Important: This will take considerable time (hours/days)
mdadm --grow /dev/md0 --level=6 --raid-devices=6 --backup-file=/root/md0_backup

# Monitor progress (watch every few hours)
watch cat /proc/mdstat

# Alternative monitoring method
mdadm --detail /dev/md0 | grep -i progress

3. Filesystem Expansion


# For ext4 filesystems (common case)
resize2fs /dev/md0p1

# For XFS filesystems
xfs_growfs /mnt/array

After conversion, consider these optimizations:


# Update stripe cache size (improves performance)
echo 32768 > /sys/block/md0/md/stripe_cache_size

# Consider adding write-intent bitmap
mdadm --grow /dev/md0 --bitmap=internal

# Monitor performance with iostat during rebuild
iostat -x 1

Failed Disk During Conversion


# If a disk fails mid-operation
mdadm --manage /dev/md0 --fail /dev/sdX
mdadm --manage /dev/md0 --remove /dev/sdX

# Add replacement disk
mdadm --add /dev/md0 /dev/sdY

Stalled Reshape Operation


# Check kernel messages
dmesg | grep md

# If necessary, restart the reshape
mdadm --assemble --update=resync /dev/md0

For those preferring a cleaner method (though requires temporary space):


# Create new RAID6 array
mdadm --create /dev/md1 --level=6 --raid-devices=6 /dev/sd{a..f}

# Copy data while old array is mounted
rsync -aHAXv --progress /mnt/old_array/ /mnt/new_array/

# Update fstab and grub before rebooting

When expanding a storage array, converting from RAID5 to RAID6 offers improved fault tolerance while maintaining available capacity. With the scenario of migrating from 3 disks to 6 disks in a Linux software RAID (mdadm) setup, we need a method that preserves existing data while restructuring the array.

  • Full backup of critical data (even though we won't need to restore it)
  • All new disks properly installed and recognized by the system
  • Current RAID5 array in healthy state (check with cat /proc/mdstat)
  • Enough free space on temporary storage if using intermediate steps

1. First, grow the existing RAID5 array with new disks:


# Add new disks one by one:
mdadm --add /dev/mdX /dev/sdX1
mdadm --add /dev/mdX /dev/sdX2
mdadm --add /dev/mdX /dev/sdX3

# Grow the array to use all 6 disks:
mdadm --grow /dev/mdX --raid-devices=6

2. Change the RAID level from 5 to 6:


mdadm --grow /dev/mdX --level=6 --backup-file=/root/mdX_backup

The backup file is crucial as it stores critical metadata during conversion. Place it on separate storage if possible.

The conversion process can take significant time (potentially days for multi-TB arrays). Consider these optimizations:


# Increase rebuild speed (adjust based on system load):
echo 50000 > /proc/sys/dev/raid/speed_limit_min
echo 200000 > /proc/sys/dev/raid/speed_limit_max

# Check progress:
cat /proc/mdstat
watch -n 5 cat /proc/mdstat

After completion, verify the new array configuration:


mdadm --detail /dev/mdX

# Expected output should show:
# Version : 1.2
# Raid Level : raid6
# Total Devices : 6
# Active Devices : 6

Update your mdadm.conf to persist the configuration:


mdadm --detail --scan >> /etc/mdadm/mdadm.conf

For larger arrays where direct conversion might be risky, consider this method:


# Create new RAID6 with 4 disks (2 for parity, 2 for data):
mdadm --create /dev/mdY --level=6 --raid-devices=4 /dev/sd[b-e]1

# Copy data from old array:
rsync -avxHAX --progress /mount/point/old_array/ /mount/point/new_array/

# Then expand with remaining disks:
mdadm --add /dev/mdY /dev/sdf1
mdadm --add /dev/mdY /dev/sdg1
mdadm --grow /dev/mdY --raid-devices=6

Use these commands to monitor the process:


# Detailed array information:
mdadm --detail /dev/mdX

# Event log:
dmesg | grep md

# Check individual disk health:
smartctl -a /dev/sdX

If the process fails, consult the backup file and mdadm man pages for recovery options.