How to Trigger a Command After ZFS Scrub Completion: A Cron-Based Monitoring Solution


4 views

When automating ZFS pool maintenance, many administrators want to execute follow-up actions after scrub operations complete. The core challenge lies in detecting the state transition from "scrubbing" to "completed" programmatically.

The most reliable method involves parsing zpool status output. Here's how to check scrub status in a script:

#!/bin/bash
POOL="tank"
STATUS=$(zpool status $POOL)

if [[ $STATUS == *"scrub in progress"* ]]; then
    echo "Scrub running"
elif [[ $STATUS == *"scrub completed"* ]]; then
    echo "Scrub finished"
    # Trigger your post-scrub command here
else
    echo "No scrub activity detected"
fi

For cron-based monitoring, create a wrapper script that:

  1. Initiates scrubs when needed
  2. Monitors active scrubs
  3. Executes post-scrub actions
#!/bin/bash
POOL="tank"
LOCKFILE="/var/run/zfs_scrub_monitor.$POOL.lock"

# Avoid multiple concurrent instances
if [ -e "$LOCKFILE" ]; then
    exit 0
fi
touch "$LOCKFILE"

# Check if scrub needed (older than 30 days)
LAST_SCRUB=$(zpool status $POOL | grep -oP "scrub.*completed on \K[^\n]+")
if [[ -n "$LAST_SCRUB" ]]; then
    LAST_SCRUB_TS=$(date -d "$LAST_SCRUB" +%s)
    NOW_TS=$(date +%s)
    DAYS_SINCE=$(( (NOW_TS - LAST_SCRUB_TS) / 86400 ))
    
    if [ "$DAYS_SINCE" -ge 30 ]; then
        zpool scrub "$POOL"
    fi
fi

# Monitor active scrub
while true; do
    STATUS=$(zpool status "$POOL")
    if [[ $STATUS != *"scrub in progress"* ]]; then
        if [[ $STATUS == *"scrub completed"* ]]; then
            # Scrub finished - send report
            zpool status "$POOL" | mail -s "ZFS Scrub Report for $POOL" admin@example.com
        fi
        break
    fi
    sleep 3600 # Check hourly
done

rm -f "$LOCKFILE"

For systems using systemd, you can create a service unit that triggers after scrub completion:

[Unit]
Description=Post-ZFS Scrub Handler
After=zfs-scrub@tank.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/post-scrub-handler.sh

[Install]
WantedBy=multi-user.target

Always include proper error checking in your scripts:

if ! zpool status "$POOL" &>/dev/null; then
    echo "Error: Pool $POOL not found" >&2
    exit 1
fi

For large pools, consider these monitoring improvements:

  • Use zpool status -p for parseable timestamps
  • Log scrub progress to a temporary file
  • Implement exponential backoff in your polling intervals

ZFS scrubs operate asynchronously by design, which creates challenges for automation. When you initiate a scrub with zpool scrub $POOL, the command returns immediately while the actual operation continues in the background. This behavior makes it difficult to chain follow-up commands directly.

The most reliable method involves polling zpool status and parsing its output. Here's a bash script that implements this approach:

#!/bin/bash
POOL="tank"
STATUS_FILE="/tmp/zfs_scrub_status.last"

current_status=$(zpool status $POOL | grep -E "scrub in progress|scan: scrub")
previous_status=$(cat $STATUS_FILE 2>/dev/null)

if [[ "$current_status" != "$previous_status" ]]; then
    # Status changed
    if [[ "$previous_status" == *"scrub in progress"* && "$current_status" != *"scrub in progress"* ]]; then
        # Scrub completed
        zpool status $POOL | mail -s "ZFS Scrub Report for $POOL" admin@example.com
    fi
    
    # Update status file
    echo "$current_status" > $STATUS_FILE
fi

For systems using systemd, you can create a more elegant solution:

# /etc/systemd/system/zfs-scrub-monitor.service
[Unit]
Description=ZFS Scrub Completion Monitor
After=zfs.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/zfs-scrub-monitor.sh
# /etc/systemd/system/zfs-scrub-monitor.timer
[Unit]
Description=Run ZFS scrub monitor every 5 minutes

[Timer]
OnCalendar=*:0/5

[Install]
WantedBy=timers.target

For real-time monitoring without polling, you can use the ZFS Event Daemon (ZED). Create a file at /etc/zfs/zed.d/scrub-complete.sh:

#!/bin/sh
[ "$ZEVENT_SUBCLASS" = "scrub_finish" ] || exit 0

zpool status "$ZEVENT_POOL" | mail -s "ZFS Scrub Completed: $ZEVENT_POOL" admin@example.com

Remember to make it executable with chmod +x and restart ZED.

For traditional cron users, this entry would check every 10 minutes:

*/10 * * * * /usr/local/bin/zfs-scrub-monitor.sh

Here's a comprehensive script that handles both initiating scrubs and monitoring completion:

#!/bin/bash
POOL="tank"
LOCK_FILE="/var/run/zfs_scrub.lock"
LOG_FILE="/var/log/zfs_scrub.log"

# Only run if not already scrubbing
if zpool status $POOL | grep -q "scrub in progress"; then
    exit 0
fi

# Check if we should initiate a new scrub
if [[ ! -f $LOCK_FILE ]]; then
    # Start new scrub
    echo "Initiating scrub on $(date)" >> $LOG_FILE
    zpool scrub $POOL
    touch $LOCK_FILE
else
    # Check if scrub completed
    if ! zpool status $POOL | grep -q "scrub in progress"; then
        echo "Scrub completed on $(date)" >> $LOG_FILE
        zpool status $POOL >> $LOG_FILE
        zpool status $POOL | mail -s "ZFS Scrub Completed: $POOL" admin@example.com
        rm -f $LOCK_FILE
    fi
fi