During late-night debugging sessions, I've frequently encountered situations where tail -f application.log
suddenly stops displaying new entries - even though the log file continues growing. The classic Ctrl+C and restart approach works, but let's understand why this happens.
The -f
flag makes tail track the file descriptor, not just the filename. Here's what happens at the system level:
// Simplified sequence
1. Opens file (gets file descriptor)
2. Seeks to end position
3. Periodically checks for new data
4. Displays new content
Inode Changes: When log rotation occurs (even if the filename stays the same), the original inode gets replaced. Example rotation scenarios:
$ ls -i application.log
123456 application.log # Original inode
$ logrotate --force /etc/logrotate.conf
$ ls -i application.log
789012 application.log # New inode
NFS/Special Filesystems: Network or virtual filesystems may cache metadata differently.
Option 1: Use --follow=name
tail --follow=name application.log # Tracks filename rather than descriptor
Option 2: Use inotifywait
while true; do
inotifywait -e modify application.log
tail -n 50 application.log
done
Option 3: The Nuclear Approach
watch -n 1 "tail -n 50 application.log"
For cases where the file isn't rotating but tail still freezes:
# Check for file system issues
df -h /var/log
mount | grep log
# Verify inode hasn't changed
stat -c %i application.log
For critical systems, consider dedicated tools:
# multitail for multiple logs
multitail -cS app_log /var/log/app/*.log
# lnav for advanced parsing
lnav /var/log/app/
Every sysadmin and developer has relied on tail -f
at some point, but this trusty tool can sometimes fail us. On RHEL 5.2 x64 systems, you might observe:
# Starts working fine
$ tail -f /var/log/app/error.log
[2023-01-01 12:00] Error: Connection timeout
[2023-01-01 12:01] Warning: High memory usage
# Then suddenly stops updating
# ...while the log file continues growing
While log rotation is the usual suspect (where tail -f
fails to track moved/renamed files), this particular issue occurs without rotation. Key observations:
- The inode remains unchanged (
ls -i /path/to/log
shows same inode) - File continues growing (
du -h
shows increasing size) - Restarting
tail -f
temporarily fixes the problem
Linux uses file descriptors to track open files. The tail -f
command typically works by:
- Opening the file and remembering its inode
- Periodically checking for new content (using
inotify
on modern kernels) - Reading new data when detected
On RHEL 5.2, several factors can break this flow:
# Check current inotify watches (modern systems)
$ cat /proc/sys/fs/inotify/max_user_watches
# On RHEL 5.2, watch limitations might apply differently
Here are several approaches that have worked for engineers:
# 1. Use --follow=name (explicitly tracks filename)
tail --follow=name /var/log/app.log
# 2. Use the BSD-style -F flag (equivalent to --follow=name --retry)
tail -F /var/log/app.log
# 3. Alternative tools with better tracking:
less +F /var/log/app.log # Press Ctrl+C then F to restart
multitail /var/log/app.log
# 4. For persistent monitoring:
while true; do tail -n 50 /var/log/app.log; sleep 1; done
If you need bulletproof logging and can modify the application:
# Create a named pipe
mkfifo /var/log/app.pipe
# Configure app to write to pipe
# Then read from two directions:
tail -f /var/log/app.pipe & # Reads from pipe
tail -f /var/log/app.log # Reads from regular file
Remember that RHEL 5.2 is quite old (released 2008) - consider upgrading if possible, as newer kernels handle file tracking more reliably.