Debugging Why “tail -f” Stops Updating Logs: File Rotation, Inode Changes, and Solutions


9 views

During late-night debugging sessions, I've frequently encountered situations where tail -f application.log suddenly stops displaying new entries - even though the log file continues growing. The classic Ctrl+C and restart approach works, but let's understand why this happens.

The -f flag makes tail track the file descriptor, not just the filename. Here's what happens at the system level:

// Simplified sequence
1. Opens file (gets file descriptor)
2. Seeks to end position
3. Periodically checks for new data
4. Displays new content

Inode Changes: When log rotation occurs (even if the filename stays the same), the original inode gets replaced. Example rotation scenarios:

$ ls -i application.log
123456 application.log  # Original inode
$ logrotate --force /etc/logrotate.conf
$ ls -i application.log
789012 application.log  # New inode

NFS/Special Filesystems: Network or virtual filesystems may cache metadata differently.

Option 1: Use --follow=name

tail --follow=name application.log  # Tracks filename rather than descriptor

Option 2: Use inotifywait

while true; do
  inotifywait -e modify application.log
  tail -n 50 application.log
done

Option 3: The Nuclear Approach

watch -n 1 "tail -n 50 application.log"

For cases where the file isn't rotating but tail still freezes:

# Check for file system issues
df -h /var/log
mount | grep log

# Verify inode hasn't changed
stat -c %i application.log

For critical systems, consider dedicated tools:

# multitail for multiple logs
multitail -cS app_log /var/log/app/*.log

# lnav for advanced parsing
lnav /var/log/app/

Every sysadmin and developer has relied on tail -f at some point, but this trusty tool can sometimes fail us. On RHEL 5.2 x64 systems, you might observe:

# Starts working fine
$ tail -f /var/log/app/error.log
[2023-01-01 12:00] Error: Connection timeout
[2023-01-01 12:01] Warning: High memory usage

# Then suddenly stops updating
# ...while the log file continues growing

While log rotation is the usual suspect (where tail -f fails to track moved/renamed files), this particular issue occurs without rotation. Key observations:

  • The inode remains unchanged (ls -i /path/to/log shows same inode)
  • File continues growing (du -h shows increasing size)
  • Restarting tail -f temporarily fixes the problem

Linux uses file descriptors to track open files. The tail -f command typically works by:

  1. Opening the file and remembering its inode
  2. Periodically checking for new content (using inotify on modern kernels)
  3. Reading new data when detected

On RHEL 5.2, several factors can break this flow:

# Check current inotify watches (modern systems)
$ cat /proc/sys/fs/inotify/max_user_watches

# On RHEL 5.2, watch limitations might apply differently

Here are several approaches that have worked for engineers:

# 1. Use --follow=name (explicitly tracks filename)
tail --follow=name /var/log/app.log

# 2. Use the BSD-style -F flag (equivalent to --follow=name --retry)
tail -F /var/log/app.log

# 3. Alternative tools with better tracking:
less +F /var/log/app.log  # Press Ctrl+C then F to restart
multitail /var/log/app.log

# 4. For persistent monitoring:
while true; do tail -n 50 /var/log/app.log; sleep 1; done

If you need bulletproof logging and can modify the application:

# Create a named pipe
mkfifo /var/log/app.pipe

# Configure app to write to pipe
# Then read from two directions:
tail -f /var/log/app.pipe &    # Reads from pipe
tail -f /var/log/app.log       # Reads from regular file

Remember that RHEL 5.2 is quite old (released 2008) - consider upgrading if possible, as newer kernels handle file tracking more reliably.