Journald vs Rsyslog: Resolving Log Duplication and Understanding Data Sources in CentOS 7


2 views

On modern CentOS/RHEL 7+ systems, we often observe log messages appearing both in journald and traditional syslog files like /var/log/messages. This occurs because:

# Default journald.conf setting that enables forwarding
ForwardToSyslog=yes  # Even when set to no, duplication may persist

The rsyslog configuration you shared reveals important mechanics:

# Traditional Unix socket logging (mostly disabled)
$ModLoad imuxsock  
$OmitLocalLogging on

# Primary journald integration
$ModLoad imjournal

Key points about data sources:

  • imjournal module: Primary path for rsyslog to read journald entries
  • imuxsock: Legacy path for direct application logging (largely disabled)
  • Some services may still log directly via /dev/log socket

To eliminate duplication while maintaining all log sources:

# In /etc/systemd/journald.conf
ForwardToSyslog=no

# In /etc/rsyslog.conf
# Ensure proper journal integration
module(load="imjournal" StateFile="imjournal.state")

# For applications still using legacy logging
input(type="imuxsock" Socket="/dev/log" CreatePath="on")

Check log sources with these commands:

# View journald-originating messages in rsyslog
journalctl -u rsyslog --no-pager | grep imjournal

# Detect direct syslog senders
lsof | grep /dev/log

The imjournal module maintains its own cursor state in /var/lib/rsyslog/imjournal.state. For high-volume systems:

# Optimize polling interval (default 10 seconds)
$IMJournalRatelimitInterval 30
$IMJournalRatelimitBurst 50000

On modern CentOS/RHEL 7+ systems, we see both journald and rsyslog processing the same messages due to their default integration. The key configuration elements causing this are:

# Default journald.conf setting causing forwarding
[Journal]
ForwardToSyslog=yes

# Rsyslog's journal integration
$ModLoad imjournal
$OmitLocalLogging on

Contrary to initial assumptions, rsyslog in this configuration isn't just mirroring journald. It handles:

  • Kernel messages via imklog (if loaded)
  • Legacy applications using direct syslog() calls
  • Network syslog messages (when configured)

For high-volume systems, running both services creates:

# Measure journald impact
journalctl --disk-usage
# Check rsyslog resource usage
ps -eo pid,cmd,%mem,%cpu | grep rsyslog

Option 1: Full rsyslog replacement

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
ForwardToSyslog=no

# Then disable rsyslog:
systemctl disable --now rsyslog

Option 2: Hybrid approach

# Keep rsyslog only for specific facilities
if $programname != 'systemd' then {
    *.emerg /var/log/critical.log
    authpriv.* /var/log/secure
}

After making changes, test with:

# Generate test messages
logger "rsyslog test"
systemd-cat echo "journald test"

# Verify capture
journalctl -b | grep "test"
grep "test" /var/log/messages

Remember that journald stores structured data including metadata fields that rsyslog might drop. Compare captures:

journalctl -o json-pretty -n1
vs
grep -m1 . /var/log/messages