When working with embedded Linux systems like CentOS 6.2 in headless appliances, performing filesystem checks becomes particularly tricky. The standard fsck utility refuses to operate on mounted filesystems, even when they're mounted read-only - a common safety measure that ironically prevents recovery operations.

The usual approaches don't work in this scenario:

• fsck -f only forces checks on clean unmounted filesystems
• Interactive prompts don't work in headless environments
• Remounting as read-only using mount -o remount,ro often fails due to busy filesystems

Here are several approaches I've tested in embedded environments:

1. Using SysRq Magic Commands

The most reliable method I've found for remounting as read-only:

echo s > /proc/sysrq-trigger
echo s > /proc/sysrq-trigger
echo u > /proc/sysrq-trigger

This forces synchronization and remounting as read-only, though fsck may still see it as mounted.

2. Forcing fsck with -n Option

While not ideal for repairs, this at least lets you check filesystem status:

fsck -n /dev/sda1

3. Compile-Time Solutions

For persistent embedded systems, consider patching e2fsprogs to:

• Remove the mounted filesystem check
• Add a --force-mounted flag
• Default to read-only operations when forced

Here's a sample patch for e2fsck:

diff --git a/e2fsck/unix.c b/e2fsck/unix.c
index abc1234..def5678 100644
--- a/e2fsck/unix.c
+++ b/e2fsck/unix.c
@@ -1234,7 +1234,7 @@ static void check_mount(e2fsck_t ctx)
        char *cp;
        int mount_flags;
 
-       if (ctx->options & E2F_OPT_READONLY)
+       if ((ctx->options & E2F_OPT_READONLY) || (ctx->options & E2F_OPT_FORCE))
                return;
 
        if (ext2fs_check_mount_point(ctx->device_name, &mount_flags,

For production systems where modifying fsck isn't desirable:

1. Temporary Umount Workaround

umount -l /mnt/point
fsck /dev/sdX1
mount -o remount,rw /mnt/point

2. Using Emergency Shell

Configure your recovery tool to:

#!/bin/bash
exec /dev/tty1 2>&1
setsid /bin/bash -c "exec /bin/bash /dev/tty1 2>&1"
echo "Remounting filesystem readonly..."
mount -o remount,ro /
echo "Forcing filesystem check..."
fsck -fy /dev/sda1
echo "Rebooting..."
reboot -f

Remember that forcing fsck on mounted filesystems can lead to:

• Data corruption if any process has pending writes
• Inconsistent results due to cached data
• Unexpected behavior with journaling filesystems

Always test thoroughly in your specific environment before deploying to production.

---

When working with headless embedded Linux systems (like CentOS 6.2 appliances), administrators often face the dilemma of filesystem checks on mounted partitions. While remounting as read-only through sysrq-trigger makes the filesystem theoretically safe for checking, standard fsck implementations still refuse to operate due to the mounted status.

The default behavior stems from:

• Kernel caching mechanisms that can lead to inconsistencies
• Potential journaling conflicts with ext3/4 filesystems
• Lack of terminal confirmation in non-interactive environments

Here are three proven approaches:

Method 1: The -n "Dry Run" Workaround

# First remount as read-only
echo s > /proc/sysrq-trigger
echo s > /proc/sysrq-trigger  
echo u > /proc/sysrq-trigger

# Then force check with -n (no-op) followed by -f
e2fsck -n /dev/sda1
e2fsck -f -p /dev/sda1

Method 2: Compile a Modified e2fsck

For embedded systems where you control the toolchain:

# In e2fsprogs source, modify lib/e2fsck/problem.c:
- if (ctx->flags & E2F_FLAG_READONLY) {
+ if (0) { // Bypass mounted check

Method 3: Temporary Loop Device

# For ext4 partitions:
losetup --find --show --read-only /dev/sda1
e2fsck -f /dev/loop0
losetup -d /dev/loop0

When implementing forced fsck:

• Always have a backup mechanism (like dd if=/dev/sda1 of=/backup.img)
• Monitor kernel messages with dmesg -w during operation
• Consider using fsck.mode=force in kernel boot parameters for future boots

For our CentOS 6.2 appliance, we implemented this recovery sequence:

#!/bin/bash
# Emergency filesystem check routine
sync; sync; sync
echo s > /proc/sysrq-trigger
echo s > /proc/sysrq-trigger
echo u > /proc/sysrq-trigger
/sbin/e2fsck -f -y -C 0 /dev/mmcblk0p2 || {
    logger -t recovery "FS check failed"
    /sbin/reboot -f
}
/sbin/reboot