Hardening ext4 Filesystem Against Power Loss Corruption on Embedded Linux Systems


2 views

Industrial flash-based IDE modules are particularly susceptible to filesystem corruption during sudden power outages. The ext4 journaling filesystem, while robust, still requires proper configuration for embedded environments where power stability isn't guaranteed. From your dumpe2fs output, we can see several potential optimization points that aren't currently configured.

# Critical parameters from your current setup:
Filesystem state:         not clean
Errors behavior:          Continue
Default mount options:    (none)
Journal backup:           inode blocks

First, let's implement the write cache disablement you mentioned. For IDE flash modules, we need to operate at both the filesystem and block device levels:

# Disable write caching on the block device
hdparm -W0 /dev/sda

# Verify the setting
hdparm -W /dev/sda

# Permanent solution - add to rc.local:
echo 'hdparm -W0 /dev/sda' >> /etc/rc.local

Your current mount options show "(none)" - this is where we can make significant improvements. Modify your /etc/fstab entry:

/dev/sda1 / ext4 noatime,nodiratime,data=journal,barrier=1,commit=30,errors=remount-ro 0 1

Key options explained:

  • data=journal: Full journaling including data (more secure but slightly slower)
  • barrier=1: Ensures proper write ordering
  • commit=30: Reduces metadata sync frequency
  • errors=remount-ro: Safer than continuing on errors

When reformatting (recommended for clean implementation), use these optimized parameters:

mkfs.ext4 -O ^has_journal,^extent,^flex_bg -E stride=4,stripe_width=4 -b 4096 -L VideoServer /dev/sda1
tune2fs -o journal_data /dev/sda1
tune2fs -c 0 -i 0 /dev/sda1  # Disable time-based checks

For your 4GB flash module, we should optimize the journal size and behavior:

# Reduce journal size to 16MB (appropriate for your storage size)
tune2fs -J size=16 /dev/sda1

# Enable writeback mode for non-critical data
tune2fs -o journal_data_writeback /dev/sda1

# Ensure journal checksumming is enabled
tune2fs -O journal_checksum /dev/sda1

For deployments where manual fsck isn't possible, implement automatic recovery:

# In /etc/default/rcS add:
FSCKFIX=yes

# Create emergency recovery script /usr/local/bin/auto_fsck:
#!/bin/sh
if [ -e /forcefsck ]; then
    echo "Running forced filesystem check"
    fsck -y /dev/sda1
    reboot
fi

# Make executable and add to crontab:
chmod +x /usr/local/bin/auto_fsck
(crontab -l ; echo "@reboot /usr/local/bin/auto_fsck") | crontab -

Configure these kernel parameters in /etc/sysctl.conf:

vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.dirty_expire_centisecs = 3000
vm.swappiness = 0

These settings reduce filesystem caching and prioritize writing changes to disk more frequently.

Implement regular filesystem health checks:

# Weekly filesystem check script
#!/bin/bash
tune2fs -l /dev/sda1 | grep -i 'last checked' > /var/log/fscheck.log
smartctl -H /dev/sda >> /var/log/fscheck.log

Combine these measures with proper power management at the hardware level (UPS or supercapacitors if possible) for maximum reliability in industrial environments.

Industrial deployments using flash-based storage frequently face filesystem corruption during unexpected power outages. My experience with 4GB IDE flash modules running EXT4 shows this isn't just an inconvenience - it can render systems unbootable until manual fsck intervention.

The dumpe2fs output reveals several risk factors:

Filesystem state: not clean
Last write time: Mon Oct 3 16:34:01 2011
Mount count: 2
Maximum mount count: 26

Key vulnerabilities emerge from:

  • Write caching in both kernel and drive controllers
  • Default EXT4 journaling behavior
  • Delayed allocation features

First, disable write caching at multiple levels:

# Disable drive write cache
hdparm -W0 /dev/sda

# Mount with explicit sync options
mount -o sync,data=journal /dev/sda1 /mnt

# Add to fstab for persistence:
/dev/sda1 / ext4 sync,data=journal,noatime,nodiratime,errors=remount-ro 0 1

Reformat with robustness-focused parameters:

mkfs.ext4 -O ^has_journal -E stride=16,stripe-width=16 /dev/sda1
tune2fs -c 0 -i 0 /dev/sda1

This configuration:

  • Removes journaling (controversial but effective)
  • Aligns to flash erase blocks
  • Disables forced fsck intervals

Add these kernel boot parameters:

rootflags=sync elevator=noop libata.force=noncq

For embedded builds, consider patching the kernel with:

CONFIG_EXT4_FS_POSIX_ACL=n
CONFIG_EXT4_FS_SECURITY=n
CONFIG_EXT4_DEBUG=n

For read-mostly systems, implement OverlayFS:

mount -t overlay overlay -o lowerdir=/,upperdir=/mnt/upper,workdir=/mnt/work /new_root

This creates a write-back cache that's trivial to reconstruct after power loss.

Implement emergency recovery scripts in initramfs:

#!/bin/sh
fsck -y /dev/sda1
mount -o remount,rw /dev/sda1 /
[ -x /etc/init.d/checkfs ] && /etc/init.d/checkfs

Log filesystem events with:

tune2fs -o journal_data_writeback /dev/sda1
debugfs -w /dev/sda1 -R "set_super_value mount_opts debug"