Recently encountered a perplexing situation where rsyslog on a Debian server stopped writing to log files despite the daemon running normally. The symptoms:

# ls -l /var/log/*.log
-rw-r--r-- 1 root root 0 Jun 27 00:25 /var/log/alternatives.log
-rw-r----- 1 root adm  0 Jun 26 13:03 /var/log/auth.log
[...]

First, verify basic functionality with the logger command:

# logger "test message"
# tail -n 1 /var/log/syslog
(no output)

The lsof output reveals rsyslog isn't opening any log files:

# lsof -p $(pgrep rsyslog)
[...]
rsyslogd 1855 root    0u  unix 0xebe8e800      0t0        640 /dev/log
rsyslogd 1855 root    5r   REG        0,3        0 4026532176 /proc/kmsg

Check if rsyslog is actually processing rules:

# rsyslogd -N1
rsyslogd: version 8.4.2, config validation run...
rsyslogd: End of config validation run. Bye.

Verify working directory permissions:

# ls -ld /var/log/
drwxr-xr-x 10 root root 4096 Jun 28 10:00 /var/log/

While the root partition shows 98% usage, /var/log typically resides there:

# df -h /var/log
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   22G  629M  98% /

Try creating manual log entries:

# echo "$(date) Test message" >> /var/log/syslog
# ls -l /var/log/syslog
-rw-r----- 1 root adm 25 Jun 28 10:05 /var/log/syslog

Use strace to monitor file operations:

# strace -f -p $(pgrep rsyslog) -e open,write,close
[pid 28824] open("/var/log/auth.log", O_WRONLY|O_CREAT|O_APPEND, 0640) = -1 ENOSPC (No space left on device)

The strace output shows ENOSPC errors when rsyslog attempts to write to log files. Despite some available space, the system's inode table may be full:

# df -i
Filesystem       Inodes  IUsed  IFree IUse% Mounted on
/dev/root       1572864 1572864     0   100% /

First, clear some inodes by removing temporary files:

# find /tmp -type f -mtime +30 -delete
# find /var/log -name "*.gz" -delete

Then restart rsyslog and verify:

# systemctl restart rsyslog
# logger "test message after cleanup"
# tail -n 1 /var/log/syslog
Jun 28 10:15:25 hostname root: test message after cleanup

For long-term maintenance, consider implementing log rotation policies:

# cat /etc/logrotate.d/custom
/var/log/syslog {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
}

---

When examining the rsyslog service status, you might see the daemon running normally but failing to write to any log files. Key observations include:

# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2023-06-26 13:03:45 UTC; 2 days ago

Yet all log files show zero bytes, even after manual rotation or using the logger command.

Begin troubleshooting with these essential commands:

# Check current rsyslog configuration
rsyslogd -N1

# Verify disk space and inodes
df -h
df -i

# Check file permissions
ls -la /var/log/ | grep '.log'

Based on your system information, several possibilities emerge:

1. Disk Space Exhaustion: Your root filesystem shows 98% usage (22GB/24GB used)
2. Permission Issues: The adm group needs write access to log files
3. Misconfigured Modules: The imuxsock module might not be loading properly

Try this comprehensive fix sequence:

# Clear disk space (critical for logging to resume)
sudo apt-get clean
sudo journalctl --vacuum-size=100M

# Reset log file permissions
sudo chown root:adm /var/log/*.log
sudo chmod 640 /var/log/*.log

# Restart rsyslog with debug mode
sudo systemctl stop rsyslog
sudo rsyslogd -dn &  # Run in foreground with debug

Verify your rsyslog.conf contains these essential directives:

module(load="imuxsock") # provides support for local system logging
module(load="imklog")   # provides kernel logging support

auth,authpriv.*         /var/log/auth.log
*.*;auth,authpriv.none  -/var/log/syslog

As a last resort, completely reset rsyslog:

sudo apt-get purge rsyslog
sudo rm -rf /etc/rsyslog.d/*
sudo apt-get install rsyslog
sudo systemctl restart rsyslog

Remember to monitor disk space usage proactively to prevent recurrence. Consider implementing logrotate with compression for long-term maintenance.