Best Practices for Expanding Linux Software RAID 5 with XFS Filesystem on Ubuntu


2 views

When creating a scalable storage solution with Linux software RAID, proper initial setup is crucial for future expansion. Here's the optimal way to create a 3-drive RAID 5 array:

# Partition each drive (repeat for /dev/sdb, /dev/sdc, /dev/sdd)
sudo fdisk /dev/sdb
# Command sequence: n → p → 1 → [enter] → [enter] → t → fd → w

# Create RAID 5 array
sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 \
--bitmap=internal --assume-clean

# Set up LVM
sudo pvcreate /dev/md0
sudo vgcreate vg_raid /dev/md0
sudo lvcreate -l 100%FREE -n lv_raid vg_raid

# Format with XFS (using optimal parameters for RAID)
sudo mkfs.xfs -f -d su=256k,sw=3 /dev/vg_raid/lv_raid

Proper mount options significantly impact performance and reliability:

# Create mount point
sudo mkdir -p /mnt/raid

# Add to fstab (with optimized XFS mount options)
echo "/dev/mapper/vg_raid-lv_raid /mnt/raid xfs defaults,noatime,nodiratime,logbufs=8,logbsize=256k 0 2" | sudo tee -a /etc/fstab

# Mount and verify
sudo mount -a
df -h /mnt/raid

The expansion process requires careful sequencing:

# Prepare the new drive (/dev/sde in this example)
sudo fdisk /dev/sde
# Same partition commands as initial setup

# Add to array and expand
sudo mdadm --add /dev/md0 /dev/sde1
sudo mdadm --grow /dev/md0 --raid-devices=4

# Monitor rebuild progress (run in separate terminal)
watch -n 60 cat /proc/mdstat

# After rebuild completes, expand LVM
sudo pvresize /dev/md0
sudo lvextend -l +100%FREE /dev/vg_raid/lv_raid
sudo xfs_growfs /mnt/raid
  • Stripe Alignment: Always use -d su=256k,sw=3 when creating XFS for RAID 5
  • Write Cache: Consider --write-behind=2048 for mdadm if using battery-backed cache
  • Monitoring: Set up email alerts in /etc/mdadm/mdadm.conf
  • Regular Checks: Schedule mdadm --check /dev/md0 via cron

When expansion fails, try these diagnostic steps:

# Check array status
sudo mdadm --detail /dev/md0

# Verify LVM information
sudo pvdisplay
sudo vgdisplay
sudo lvdisplay

# Check XFS health
sudo xfs_repair -n /dev/vg_raid/lv_raid

# Examine kernel messages
dmesg | grep -i md

For more flexible management, consider this LVM-first approach:

# Create RAID 1 for each pair (better for expansion)
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
sudo pvcreate /dev/md0
sudo vgcreate vg_raid /dev/md0

# Later expansion:
sudo mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdd1 /dev/sde1
sudo pvcreate /dev/md1
sudo vgextend vg_raid /dev/md1

When creating an expandable storage solution, the combination of Linux MDADM RAID 5, LVM, and XFS provides excellent flexibility. Here's the proper initialization sequence:

# Partition preparation (repeat for each drive)
sudo fdisk /dev/sdX <

To ensure your storage survives reboots and performs optimally:

# Update mdadm.conf
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

# Add to fstab with XFS-specific options
echo "/dev/mapper/vg_raid-lv_raid /raid xfs noatime,nodiratime,logbufs=8 0 2" | sudo tee -a /etc/fstab

# Update initramfs
sudo update-initramfs -u

The real power comes when you need to add capacity. Here's the complete expansion workflow:

# Prepare the new drive (same as initial partitioning)
sudo fdisk /dev/sde <

For optimal performance with growing RAID 5 arrays:

  • Use --bitmap=internal during creation for faster recovery
  • Set proper stripe unit (-d su=) when creating XFS filesystem
  • Schedule expansions during low-usage periods
  • Monitor /proc/mdstat during expansion operations

Essential commands for managing your expanding storage:

# Check RAID status
sudo mdadm --detail /dev/md0

# Check LVM status
sudo vgdisplay vg_raid
sudo lvdisplay /dev/vg_raid/lv_raid

# Check XFS filesystem
sudo xfs_info /raid

# Check disk health periodically
sudo smartctl -a /dev/sdX

When expanding fails or encounters problems:

  1. Verify all partitions have the correct type (Linux raid autodetect)
  2. Check dmesg for hardware errors during expansion
  3. Ensure sufficient power supply for additional drives
  4. Monitor system temperature during long rebuilds