When you see checksum errors appearing across multiple drives in your ZFS pool, it's crucial to understand their nature before making hardware decisions. The zpool status output shows:

status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.

This indicates ZFS has detected data corruption but was able to repair it using mirror redundancy. The key metrics to watch are:

• Persistent errors across multiple scrubs
• Errors appearing on multiple drives simultaneously
• Correlation with kernel I/O errors

Begin with these fundamental checks:

# Check SMART status
smartctl -a /dev/sdX

# View detailed disk errors
dmesg | grep -i error

# Check controller information
sas2flash -listall
modinfo mpt2sas

If SMART shows no issues (all tests passed, no reallocated sectors), the problem likely lies elsewhere in the I/O path.

Controller Firmware Mismatch

As in this case, mismatched controller firmware and driver versions can cause issues. The output showed:

version:        20.100.00.00  # Driver
FW Ver:        17.00.01.00    # Firmware

Solution: Always maintain firmware/driver version parity. For LSI controllers:

# Download appropriate firmware
sas2flash -f firmware.bin -b mptsas2.rom

Drive Spin-down Issues

Using hdparm to spin down drives can cause ZFS checksum errors when:

• Drives fail to wake promptly
• Controller loses communication during spin-up

Test by disabling spin-down temporarily:

hdparm -S0 /dev/sdX

Physical Connection Problems

For PCIe risers, backplanes, or cabling:

# Check for PCIe errors
lspci -vvv | grep -i error
dmesg | grep -i pci

Sector Size Alignment

While ashift=12 (4K sectors) didn't resolve this case, it's worth verifying:

# Check physical sector size
hdparm -I /dev/sdX | grep -i sector

# Create test pool with explicit ashift
zpool create -o ashift=12 testpool mirror sda sdb

Controller Replacement

When all else fails, HBA replacement may be necessary. Recommended steps:

1. Export pool properly
2. Physically swap controllers
3. Reimport pool with new device IDs

zpool export storage
# Swap hardware
zpool import -d /dev/disk/by-id storage

Implement proactive monitoring with:

# Regular scrubs
zpool scrub storage

# Automated error reporting
echo "0 0 * * * root /sbin/zpool scrub storage" > /etc/cron.d/zfs-scrub

For production systems, consider setting up email alerts for ZFS events.

---

When working with ZFS storage pools, checksum errors typically indicate data corruption or hardware issues. In your case with a mirrored pool of 8 drives, seeing multiple devices reporting CKSUM errors (even with SMART tests passing) suggests a systemic issue rather than individual drive failures.

From your troubleshooting journey, here are the critical steps to isolate the root cause:

# Check current pool status
zpool status -v

# Monitor kernel messages in real-time
dmesg -wH

# Verify drive health (example for /dev/sda)
smartctl -a /dev/sda

The most revealing finding was the controller firmware/driver mismatch. LSI/Broadcom controllers are particularly sensitive to version alignment:

# Check driver version
modinfo mpt2sas | grep version

# Verify firmware version
sas2flash -listall

Your discovery about drive spindown being a potential trigger is important. ZFS expects immediate access to drives, and aggressive power management can cause:

• Timeout-related checksum errors
• I/O errors during wake-up cycles
• False positive drive failure indications

The successful resolution with the Supermicro AOC-SAS2LP-MV8 confirms that some HBAs handle drive states better than others. When selecting a replacement controller:

1. Verify Linux driver support
2. Check for known ZFS compatibility issues
3. Ensure firmware update paths exist

After implementing fixes, establish proactive monitoring:

# Regular scrub schedule
zpool set scrub=weekly storage

# Automated error reporting
#!/bin/bash
ERRORS=$(zpool status | grep -c "errors")
if [ "$ERRORS" -gt 0 ]; then
    mail -s "ZFS Errors Detected" admin@example.com <<< "$(zpool status)"
fi

For environments where drive spindown is mandatory, consider:

• Adjusting ZFS timeout values
• Using L2ARC for frequently accessed data
• Implementing a warm standby pool

The key takeaway is that ZFS checksum errors often point to underlying hardware/configuration issues rather than actual drive failures. Systematic elimination of potential causes - from firmware versions to physical connections - is essential for reliable storage operations.