LVM on Software RAID vs. RAID on LVM: Performance and Architecture Comparison for Linux Systems


2 views

When managing storage in Linux environments, the relationship between LVM (Logical Volume Manager) and software RAID (mdadm) is critical. There are two primary approaches:


# LVM-on-RAID (Traditional approach)
Physical Disks -> RAID -> LVM -> Filesystem

# RAID-on-LVM (Alternative approach) 
Physical Disks -> LVM -> RAID -> Filesystem

LVM on Software RAID Advantages:

  • Better performance as RAID operates directly on physical devices
  • Proven stability with mdadm handling disk failures
  • Easier disk replacement in degraded arrays
  • Supports all RAID levels (0,1,5,6,10)

RAID on LVM Advantages:

  • More flexible volume management before RAID configuration
  • Easier to resize underlying physical volumes
  • Potential for more complex storage topologies

Benchmark tests consistently show better throughput with LVM-on-RAID:


# Sample benchmark command for LVM-on-RAID
hdparm -tT /dev/mapper/vg0-lv0

# Expected results on SSD RAID10:
Timing cached reads:   3000 MB in  2.00 seconds
Timing buffered reads: 1800 MB in  3.00 seconds

Here's how to properly set up LVM on software RAID:


# Create RAID array
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

# Prepare for LVM
pvcreate /dev/md0
vgcreate vg0 /dev/md0
lvcreate -L 100G -n lv0 vg0

# Filesystem creation
mkfs.ext4 /dev/vg0/lv0

This architecture might make sense when:

  • Using advanced LVM features like thin provisioning
  • Implementing complex storage tiering
  • Needing frequent volume resizing

For most production environments, I recommend:

  1. Use LVM-on-RAID for better performance and reliability
  2. Monitor RAID health with mdadm --detail /dev/mdX
  3. Implement proper LVM snapshots for backups
  4. Consider write-intent bitmaps for RAID rebuilds

When setting up storage systems in Linux environments, the relationship between LVM (Logical Volume Manager) and software RAID creates fundamental architectural decisions. The two primary approaches are:

  • LVM on top of software RAID (mdadm → LVM)
  • RAID implemented within LVM (LVM → RAID)

This remains the most battle-tested configuration in production environments:

# Creating RAID first
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1

# Then setting up LVM
pvcreate /dev/md0
vgcreate vg_raid /dev/md0
lvcreate -L 100G -n lv_data vg_raid

Key Advantages

  • Stable performance characteristics: The RAID layer handles striping/mirroring before LVM sees the storage
  • Easier recovery: MDADM tools are mature for array reconstruction
  • Clear separation of concerns: RAID handles redundancy, LVM handles volume management

Modern LVM versions support RAID types natively through device mapper:

# Direct LVM RAID1 creation
pvcreate /dev/sda1 /dev/sdb1
vgcreate vg_raid /dev/sda1 /dev/sdb1
lvcreate --type raid1 -L 100G -n lv_data vg_raid

Potential Benefits

  • Simplified stack: Fewer layers in the storage subsystem
  • Flexible reconfiguration: Easier to convert between RAID levels
  • Thin provisioning integration: Works seamlessly with LVM thin pools

Benchmark results from a 4-disk system (WD Red 4TB, XFS filesystem):

Configuration Sequential Read Random 4K Write
LVM on RAID5 420 MB/s 7800 IOPS
RAID5 via LVM 390 MB/s 7200 IOPS
LVM on RAID10 580 MB/s 14200 IOPS
RAID10 via LVM 550 MB/s 13800 IOPS

System Maintenance

With LVM-on-RAID, you'll typically:

# Check array status
mdadm --detail /dev/md0

# Then check LVM status
vgdisplay vg_raid

Whereas LVM-RAID consolidation provides:

# Single command for status
lvdisplay --maps vg_raid/lv_data

Recovery Scenarios

Disk failures behave differently:

  • Traditional MDADM: Clear degraded array warnings in syslog
  • LVM RAID: Requires checking both LVM and kernel raid events

For database workloads: Stick with mdadm RAID10 + LVM for best predictable performance

For flexible development environments: Consider LVM RAID for easier reconfiguration between RAID levels

For large storage arrays: The mdadm layer still provides more mature monitoring tools