Diagnosing and Fixing Filesystem Read-Only Issues on Ubuntu 12.04 Server


2 views

When your Ubuntu 12.04 server's filesystem suddenly switches to read-only mode, it can cripple your applications. In this case, Node.js workers and MongoDB are failing due to the filesystem restriction. The dmesg logs show JOURNAL errors and remount events, indicating deeper filesystem issues.

First, check your disk health. While SMART tools might not be available (as seen in your output), try these alternatives:

# Check for bad blocks
badblocks -v /dev/sda2

# Force a filesystem check on next reboot
touch /forcefsck
shutdown -r now

Your mount output shows:

/dev/sda2 on / type ext4 (rw,errors=remount-ro)

The errors=remount-ro parameter means the system will remount as read-only when errors occur. Consider changing this to errors=continue temporarily for debugging:

# Edit fstab
nano /etc/fstab
# Change to: /dev/sda2 / ext4 defaults,errors=continue 0 1
mount -o remount /

Since this is a VPS with RAID 10 storage, the underlying hardware might be fine. The provider's suggestion to force fsck is worth trying:

# Add fsck to fstab
nano /etc/fstab
# Change to: /dev/sda2 / ext4 defaults,errors=remount-ro,fsck.mode=force 0 1

For persistent JOURNAL errors, consider these diagnostic commands:

# Check filesystem consistency
e2fsck -f /dev/sda2

# View detailed journal info
dumpe2fs /dev/sda2 | grep -i journal

# Check for I/O errors in kernel log
dmesg | grep -i 'error\|fail\|io'

Create a watchdog script to detect read-only state and attempt recovery:

#!/bin/bash
if mount | grep 'on / .*ro,'; then
    logger "Filesystem is read-only, attempting remount"
    mount -o remount,rw /
    if [ $? -ne 0 ]; then
        logger "Remount failed, forcing fsck on next boot"
        touch /forcefsck
        reboot
    fi
fi

Add this to cron to run every 5 minutes.

If the issue persists, consider using OverlayFS for critical directories:

# Create overlay mount for /var (where MongoDB stores data)
mkdir -p /var-overlay/{upper,work}
mount -t overlay overlay -olowerdir=/var,upperdir=/var-overlay/upper,workdir=/var-overlay/work /var

When your Ubuntu 12.04 filesystem suddenly switches to read-only mode, it can cripple your server operations. Based on your description, this happens every 20-50 hours, affecting MongoDB and Node.js processes managed by forever. The dmesg logs show filesystem errors and JOURNAL issues, but the root cause isn't immediately clear.

Your VPS provider claims the RAID 10 storage is healthy, which makes hardware failure less likely. The mount output shows:

/dev/sda2 on / type ext4 (rw,errors=remount-ro)

This indicates your root partition is on sda2 with the errors=remount-ro mount option, which automatically makes the filesystem read-only when errors are detected.

First, let's verify filesystem integrity:

sudo touch /forcefsck
sudo reboot

After reboot, check the system logs for fsck results:

grep -i fsck /var/log/syslog

For a more thorough check, run:

sudo fsck -yf /dev/sda2

If you encounter "filesystem is mounted" errors, you'll need to boot from a live CD/USB.

Since SMART tools aren't available (common in virtualized environments), we need alternative approaches:

sudo apt-get install sysstat
sudo iostat -x 1 10

Look for high await values or I/O errors in the output.

Journal errors often indicate deeper problems. Check journal status with:

sudo dumpe2fs /dev/sda2 | grep -i journal

If the journal appears corrupted, consider recreating it:

sudo tune2fs -O ^has_journal /dev/sda2
sudo fsck -yf /dev/sda2
sudo tune2fs -j /dev/sda2

Set up continuous monitoring to catch issues before they escalate:

sudo apt-get install smartmontools lm-sensors
sudo sensors-detect
sudo service smartd start

Create a monitoring script:

#!/bin/bash
LOG_FILE="/var/log/fs_monitor.log"
echo "$(date) - Checking filesystem status" >> $LOG_FILE
mount | grep "ro," >> $LOG_FILE
dmesg | tail -20 >> $LOG_FILE

Add these lines to your crontab (crontab -e):

0 * * * * /usr/bin/logger "Hourly filesystem check: $(mount | grep 'ro,')"
0 3 * * * /sbin/fsck -n /dev/sda2

If the issue persists, consider these advanced options:

sudo tune2fs -c 100 /dev/sda2  # Check every 100 mounts
sudo tune2fs -i 7d /dev/sda2   # Check every 7 days

For critical systems, you might want to modify the mount options in /etc/fstab:

/dev/sda2 / ext4 defaults,errors=continue 0 1

Warning: Using errors=continue risks data corruption but prevents read-only switching.