Step-by-Step Guide: Online Capacity Expansion of ZFS RAID-Z Arrays with Disk Replacement on OpenSolaris


4 views

When working with ZFS on OpenSolaris, the beauty of RAID-Z lies in its online expandability. The process involves sequential disk replacement with automatic resilvering, allowing storage expansion without downtime. Here's what happens under the hood:

# Basic ZFS pool status check
zpool status tank

Before beginning the upgrade process, ensure:

  • All existing disks are healthy (check with zpool status)
  • The system has proper SATA/SAS controller support for hot-swap
  • New disks are equal or larger capacity than existing ones

Here's how to replace each disk sequentially:

# First, offline the disk safely
zpool offline tank c1t0d0

# Physically replace the disk (hot-swap if supported)
# Then bring the new disk online
zpool replace tank c1t0d0

# Monitor resilvering progress
zpool status -v tank

After replacing all disks in the RAID-Z array:

# Check the new available capacity
zpool list tank

# Verify auto-expansion worked
zfs get available tank

For complex scenarios with different disk sizes:

# When mixing disk sizes in replacement
zpool set autoexpand=on tank
zpool replace tank c1t0d0 /dev/rdsk/c2t1d0

Remember that ZFS will only use the smallest disk's capacity in a vdev unless all disks are upgraded to larger sizes.

If the pool doesn't automatically expand:

# Manually trigger expansion
zpool online -e tank c1t0d0

# Check for partition alignment issues
prtvtoc /dev/rdsk/c1t0d0s0




When upgrading disks in a ZFS RAID-Z configuration, the process involves sequential replacement with automatic resilvering. Here's the technical workflow:

  1. Physically connect the new 1TB drive (recommended as hot-swap if supported)
  2. Identify the target disk to replace:
    # zpool status
      pool: tank
     state: ONLINE
      scan: none requested
    config:
        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c4t0d0  ONLINE       0     0     0
            c4t1d0  ONLINE       0     0     0
            c4t2d0  ONLINE       0     0     0
            c4t3d0  ONLINE       0     0     0

Execute these commands for each disk (one at a time):

# zpool replace tank c4t0d0 c5t0d0
# zpool status -v (monitor resilver progress)
# zpool offline tank c4t0d0 (after verification)

After replacing all disks, ZFS won't automatically use the new space. To expand:

# zpool set autoexpand=on tank
# zpool online -e tank c5t0d0
# zpool online -e tank c5t1d0
(... repeat for all new disks)

Confirm the new capacity and optimize performance:

# zpool list
# zfs set recordsize=1M tank/datasets
# zpool scrub tank
  • Maintain 20% free space during replacement for optimal performance
  • Consider using zfs send | zfs receive for complete pool migration if possible
  • Monitor SMART data for new disks: # smartctl -a /dev/rdsk/c5t0d0