When dealing with ZFS storage pools, encountering permanent errors after deleting corrupted files can be particularly frustrating. The scenario typically looks like this:
$ zpool status -v
errors: Permanent errors have been detected in the following files:
zstorage:<0x9f115>
This hexadecimal reference persists even after:
- Deleting the problematic file
- Running
zpool clear
- Performing additional scrubs
ZFS keeps track of data errors through its internal error log mechanism. Even when files are deleted, the error records may remain because:
- The error tracking is stored in pool metadata
- ZFS maintains checksum verification history
- The error system is designed to persist until explicitly resolved
Method 1: Using zpool clear with error counters
First check current error counters:
$ zpool status zstorage | grep -A 3 "errors:"
Then clear the counters:
$ zpool clear -nF zstorage
$ zpool scrub zstorage
Method 2: Forcing error log reset
For stubborn cases, try exporting and reimporting the pool:
$ zpool export zstorage
$ zpool import zstorage
$ zpool clear zstorage
Method 3: Creating and removing dummy files
Sometimes creating a new file at the same inode location can help:
$ touch /zstorage/tempfile
$ rm /zstorage/tempfile
$ zpool scrub zstorage
For persistent cases where standard methods fail:
Using zdb to inspect error logs:
$ sudo zdb -uuu zstorage | grep -A 10 "error logs"
Forcing error log clearing:
$ sudo zpool set failmode=continue zstorage
$ sudo zpool clear zstorage
$ sudo zpool set failmode=wait zstorage
Checking for underlying hardware issues:
$ smartctl -a /dev/disk/by-id/ata-WDC_WD30EZRX-00DC0B0_WD-WCC1T1735698
- Always verify backups before deleting corrupted files
- Consider using
zfs send | zfs recv
to clone pools periodically - Implement regular scrubs with
zpool scrub -w zstorage
- Monitor error counters with automated scripts
Remember that while these errors appear serious, they're often just metadata artifacts when the original corrupted files have been removed. The pool remains fully functional for new data.
When dealing with ZFS pools, encountering corruption errors that persist even after deleting the affected files can be particularly frustrating. The scenario typically looks like this:
errors: Permanent errors have been detected in the following files:
zstorage:<0x9f115>
The hexadecimal reference (like 0x9f115) represents a data block that was part of the deleted file's metadata or actual content. ZFS maintains these error references because:
- The checksum mismatch was recorded in the pool's error logs
- Deleting the file doesn't automatically clear the error tracking
- The scrub operation marked this as a permanent error
Here are three approaches to resolve this issue:
Method 1: Full Pool Scrub
Initiate a manual scrub to potentially clear the error:
sudo zpool scrub zstorage
sudo zpool status -v
Method 2: Export/Import the Pool
This often clears transient error states:
sudo zpool export zstorage
sudo zpool import zstorage
Method 3: Manual Error Clearing
For stubborn cases, try resetting the error counters:
sudo zpool clear zstorage
sudo zpool set autoreplace=on zstorage
If the error persists after trying all methods, consider these steps:
- Check the physical drives for SMART errors
- Verify your ZFS version and apply updates
- Check for known bugs in your specific ZFS implementation
To avoid similar issues in the future:
- Schedule regular scrubs (e.g., monthly)
- Monitor your pool's health status
- Maintain proper backups of critical data
Remember that while these errors might look alarming, they're often just remnants of already-resolved issues. The pool's actual data integrity remains unaffected in most cases.