How to Filter and Route Rsyslog Messages by Custom Tags for Application Logging


3 views

When dealing with multiple applications logging through syslog, proper message routing becomes crucial. The standard approach using logger -t TAG creates messages containing both the tag and process ID (e.g., "TAG[1234]"). This format requires specific handling in rsyslog configuration.

Instead of direct equality comparison, use the startswith property filter for tag matching:

if $syslogtag startswith 'giomanager' then /var/log/giomanager.log
& stop

Create or modify a file in /etc/rsyslog.d/ with these contents:

# Custom log routing configuration
$template CustomLogFormat,"%timegenerated% %syslogtag% %msg%\n"

if $syslogtag startswith 'giomanager' then {
    action(type="omfile" file="/var/log/giomanager.log"
           template="CustomLogFormat")
    stop
}

if $syslogtag startswith 'worker' then {
    action(type="omfile" file="/var/log/worker.log"
           template="CustomLogFormat")
    stop
}

For more complex scenarios, consider these approaches:

# Match exact tag (including PID)
if $syslogtag == 'giomanager[' then /var/log/giomanager_exact.log

# Using regular expressions
if re_match($syslogtag, '^giomanager') then /var/log/giomanager_regex.log

To properly handle log rotation, create a logrotate configuration:

# /etc/logrotate.d/giomanager
/var/log/giomanager.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

If your filters aren't working:

  1. Check rsyslog syntax: rsyslogd -N1
  2. Enable debug logging: $DebugFile /tmp/rsyslog.debug
  3. Verify message format: logger -t testtag "Test message"; tail -n 1 /var/log/syslog

Remember to restart rsyslog after configuration changes: systemctl restart rsyslog


When working with long-running processes in Linux, proper log management becomes crucial. Many developers use the logger command with custom tags:

command | logger -t CUSTOM_TAG

This approach is superior to direct file redirection because it handles log rotation automatically and integrates with the system's logging infrastructure.

A common attempt to filter these logs might look like this in /etc/rsyslog.d/60-myfilter.conf:

if $syslogtag == 'giomanager' then /var/log/giomanager.log

This often fails because the $syslogtag property includes more than just the tag name - it contains the process ID and other metadata.

For accurate tag matching, use one of these approaches:

# Method 1: Using startswith comparison
if $syslogtag startswith 'giomanager' then /var/log/giomanager.log

# Method 2: Using regex matching
if re_match($syslogtag, 'giomanager$$[0-9]+$$:') then /var/log/giomanager.log

Here's a full working configuration that handles multiple tags:

# /etc/rsyslog.d/60-custom-tags.conf
# GioManager logs
if $syslogtag startswith 'giomanager' then /var/log/giomanager.log
& stop

# API service logs
if $syslogtag startswith 'api-service' then /var/log/api-service.log
& stop

# Cron job logs
if $syslogtag startswith 'cron-script' then /var/log/cron-scripts.log
& stop

For more complex scenarios, consider these additional techniques:

# Log severity filtering combined with tags
if $syslogtag startswith 'giomanager' and $syslogseverity <= 4 then /var/log/giomanager-error.log

# Multi-tag matching
if $syslogtag startswith 'giomanager' or $syslogtag startswith 'api-service' then /var/log/combined.log

After modifying your configuration:

sudo systemctl restart rsyslog
logger -t giomanager "Test message"
tail -f /var/log/giomanager.log

If messages don't appear, check rsyslog debug output:

sudo rsyslogd -dn | grep giomanager