How to Filter and Route Rsyslog Messages by Custom Tags for Application Logging


11 views

When dealing with multiple applications logging through syslog, proper message routing becomes crucial. The standard approach using logger -t TAG creates messages containing both the tag and process ID (e.g., "TAG[1234]"). This format requires specific handling in rsyslog configuration.

Instead of direct equality comparison, use the startswith property filter for tag matching:

if $syslogtag startswith 'giomanager' then /var/log/giomanager.log
& stop

Create or modify a file in /etc/rsyslog.d/ with these contents:

# Custom log routing configuration
$template CustomLogFormat,"%timegenerated% %syslogtag% %msg%\n"

if $syslogtag startswith 'giomanager' then {
    action(type="omfile" file="/var/log/giomanager.log"
           template="CustomLogFormat")
    stop
}

if $syslogtag startswith 'worker' then {
    action(type="omfile" file="/var/log/worker.log"
           template="CustomLogFormat")
    stop
}

For more complex scenarios, consider these approaches:

# Match exact tag (including PID)
if $syslogtag == 'giomanager[' then /var/log/giomanager_exact.log

# Using regular expressions
if re_match($syslogtag, '^giomanager') then /var/log/giomanager_regex.log

To properly handle log rotation, create a logrotate configuration:

# /etc/logrotate.d/giomanager
/var/log/giomanager.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

If your filters aren't working:

  1. Check rsyslog syntax: rsyslogd -N1
  2. Enable debug logging: $DebugFile /tmp/rsyslog.debug
  3. Verify message format: logger -t testtag "Test message"; tail -n 1 /var/log/syslog

Remember to restart rsyslog after configuration changes: systemctl restart rsyslog


When working with long-running processes in Linux, proper log management becomes crucial. Many developers use the logger command with custom tags:

command | logger -t CUSTOM_TAG

This approach is superior to direct file redirection because it handles log rotation automatically and integrates with the system's logging infrastructure.

A common attempt to filter these logs might look like this in /etc/rsyslog.d/60-myfilter.conf:

if $syslogtag == 'giomanager' then /var/log/giomanager.log

This often fails because the $syslogtag property includes more than just the tag name - it contains the process ID and other metadata.

For accurate tag matching, use one of these approaches:

# Method 1: Using startswith comparison
if $syslogtag startswith 'giomanager' then /var/log/giomanager.log

# Method 2: Using regex matching
if re_match($syslogtag, 'giomanager$$[0-9]+$$:') then /var/log/giomanager.log

Here's a full working configuration that handles multiple tags:

# /etc/rsyslog.d/60-custom-tags.conf
# GioManager logs
if $syslogtag startswith 'giomanager' then /var/log/giomanager.log
& stop

# API service logs
if $syslogtag startswith 'api-service' then /var/log/api-service.log
& stop

# Cron job logs
if $syslogtag startswith 'cron-script' then /var/log/cron-scripts.log
& stop

For more complex scenarios, consider these additional techniques:

# Log severity filtering combined with tags
if $syslogtag startswith 'giomanager' and $syslogseverity <= 4 then /var/log/giomanager-error.log

# Multi-tag matching
if $syslogtag startswith 'giomanager' or $syslogtag startswith 'api-service' then /var/log/combined.log

After modifying your configuration:

sudo systemctl restart rsyslog
logger -t giomanager "Test message"
tail -f /var/log/giomanager.log

If messages don't appear, check rsyslog debug output:

sudo rsyslogd -dn | grep giomanager