Implementing Software RAID for ESXi Datastore: LVM/mdadm Solutions on Passthrough Disks


4 views

When working with VMware ESXi in budget-constrained environments, the storage architecture becomes particularly challenging. The hypervisor's native filesystems (VMFS) don't support software RAID configurations directly, forcing administrators to explore alternative solutions.

The most viable solution involves PCI passthrough of storage controllers to a Linux VM that handles the RAID management:


# Identify available PCI devices
$ lspci -nn | grep -i storage

# Enable passthrough in ESXi host (SSH access required)
esxcli hardware pci pcipassthru set -a -e true -d 0000:03:00.0

Here's how to configure mdadm RAID on passthrough disks:


# Create RAID array (example: RAID 1)
$ mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb

# Create filesystem
$ mkfs.ext4 /dev/md0

# Persistent configuration
$ mdadm --detail --scan >> /etc/mdadm/mdadm.conf

The Linux VM needs to export the storage via NFS or iSCSI:


# Install NFS server
$ apt install nfs-kernel-server

# Configure exports
$ echo "/mnt/raid *(rw,sync,no_root_squash)" >> /etc/exports
$ systemctl restart nfs-server

Several factors impact performance in this architecture:

  • RAID write penalties (especially with RAID 5/6)
  • Network overhead when using NFS/iSCSI
  • VM memory allocation for caching

While passthrough is the most direct method, other options exist:

  1. RDM (Raw Device Mapping) with external software RAID
  2. vSAN-like solutions using multiple ESXi hosts
  3. ZFS-based storage appliances in a VM

Watch for these pitfalls:

  • PCI device resets during VM reboots
  • Disk alignment problems affecting performance
  • Monitoring challenges with nested storage

When setting up a cost-effective virtualization environment with ESXi 5.1, one major limitation becomes immediately apparent: VMware doesn't natively support software RAID solutions like mdadm or LVM at the hypervisor level. This creates a challenge for small businesses that need redundancy but can't afford expensive hardware RAID controllers or separate SAN/NAS solutions.

The most common workaround involves creating a dedicated storage VM with PCI passthrough:


# Example PCI passthrough configuration in ESXi
pciPassthru0.present = "TRUE"
pciPassthru0.deviceId = "0x1234"
pciPassthru0.vendorId = "0x5678"
pciPassthru0.systemId = "lsi_mpt_sas"

Advantages:

  • Full control over RAID configuration
  • Ability to use mature Linux software RAID tools
  • Flexibility in storage configuration

Drawbacks:

  • VM suspend/resume functionality breaks
  • Added complexity in the storage path
  • Single point of failure in the storage VM

An often overlooked approach is using Raw Device Mapping (RDM) with software RAID:


# Sample vmkfstools command for RDM creation
vmkfstools -z /vmfs/devices/disks/naa.600508b1001c0123456789abcdef12 \
/vmfs/volumes/datastore1/rdm_vmdisk.vmdk

This method allows:

  • Direct disk access while maintaining VMFS benefits
  • Better performance than full passthrough
  • Simpler management than complete PCI passthrough

Here's how to configure a basic mdadm array in a storage VM:


# Identify disks
lsblk -o NAME,SIZE,MODEL

# Create RAID array
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc

# Verify array status
cat /proc/mdstat
mdadm --detail /dev/md0

# Create filesystem
mkfs.ext4 /dev/md0

To present this back to ESXi, you'd typically use NFS or iSCSI. Here's an example targetcli configuration:


# Create iSCSI backstore
/backstores/block create name=vmstore dev=/dev/md0

# Create iSCSI target
/iscsi create iqn.2023-04.com.example:storage.vmstore

# Create LUN
/iscsi/iqn.2023-04.com.example:storage.vmstore/tpg1/luns create /backstores/block/vmstore

When using software RAID under ESXi, pay special attention to:

  • Disk alignment (use 4K or 1M boundaries)
  • VMware queue depth settings
  • Filesystem choice (XFS often performs best for VM workloads)
  • Consider adding SSD caching if possible

Remember that this setup requires special backup consideration:

  • Implement regular array consistency checks
  • Backup mdadm configuration (mdadm --detail --scan > /etc/mdadm.conf)
  • Document the exact storage architecture
  • Test restoration procedures regularly