Configuring Rsyslog for Simultaneous Log Forwarding to Multiple Remote Servers with Local Buffer Fallback


3 views

When using rsyslog's traditional forwarding syntax with multiple destinations:

*.* @@primary.example.com
*.* @@secondary.example.com

The system defaults to failover behavior where secondary only receives logs if primary fails. This isn't ideal for audit logging or redundancy scenarios where you need both servers to receive identical log streams simultaneously.

Rsyslog's action directive with queue parameters provides the proper control:

# Enable parallel forwarding to both servers
*.* {
    action(type="omfwd" target="primary.example.com" port="514" protocol="tcp"
           queue.filename="primary_q" queue.size="1000000" queue.type="LinkedList"
           action.resumeRetryCount="-1")
    action(type="omfwd" target="secondary.example.com" port="514" protocol="tcp"
           queue.filename="secondary_q" queue.size="1000000" queue.type="LinkedList"
           action.resumeRetryCount="-1")
}

For your bonus requirement of separate buffer files, we extend the configuration:

# Global queue parameters
$WorkDirectory /var/lib/rsyslog  # where to place queue files
$ActionQueueType LinkedList      # use asynchronous processing
$ActionQueueFileName server1_q   # unique filename prefix
$ActionResumeRetryCount -1       # infinite retries if server down

# First destination with dedicated queue
*.* @@server1.example.com;RSYSLOG_ForwardFormat
$ActionQueueFileName server1_q

# Second destination with separate queue
*.* @@server2.example.com;RSYSLOG_ForwardFormat
$ActionQueueFileName server2_q

When implementing parallel forwarding:

  • Monitor disk I/O as each queue maintains separate buffer files
  • Adjust queue.size based on your log volume and network reliability
  • Consider using queue.dequeueBatchSize to optimize throughput

After configuration, verify with:

rsyslogd -N1  # Configuration syntax check
systemctl restart rsyslog
logger "Test message for dual forwarding"
tail -f /var/log/remote/server1.log /var/log/remote/server2.log

For production environments, consider adding TLS encryption and rate limiting to this configuration.


When configuring rsyslog to forward logs to multiple servers using the traditional syntax:

*.* @@server1
*.* @@server2

Rsyslog treats these as failover destinations rather than parallel forwarding targets. The logs will only be sent to server2 if server1 becomes unavailable.

To achieve true simultaneous forwarding, we need to use separate action queues for each destination:

# First server
*.* {
    action(type="omfwd" target="server1" port="514" protocol="tcp"
           queue.type="linkedList" queue.spoolDirectory="/var/spool/rsyslog/server1"
           queue.filename="server1_queue" queue.maxDiskSpace="1g"
           action.resumeRetryCount="-1")
}

# Second server
*.* {
    action(type="omfwd" target="server2" port="514" protocol="tcp"
           queue.type="linkedList" queue.spoolDirectory="/var/spool/rsyslog/server2"
           queue.filename="server2_queue" queue.maxDiskSpace="1g"
           action.resumeRetryCount="-1")
}

The solution above already includes disk-assisted queues for each server. Let's examine the buffering parameters:

queue.type="linkedList"          # Enables disk-assisted memory queue
queue.spoolDirectory="/path"     # Directory for queue files
queue.filename="queue_name"      # Base name for queue files
queue.maxDiskSpace="1g"          # Maximum queue size
action.resumeRetryCount="-1"     # Infinite retries

For high-volume environments, you might want to distribute logs across servers:

module(load="omfwd")
module(load="mmsequence")

template(name="dynamicForward" type="string" string="%$!dynhost%")

*.* {
    action(type="mmsequence" mode="loadbalance" server.list=["server1","server2"])
    {
        action(type="omfwd" target="$!dynhost" port="514" protocol="tcp"
               queue.type="linkedList" queue.spoolDirectory="/var/spool/rsyslog/lb"
               queue.filename="lb_queue" queue.maxDiskSpace="2g")
    }
}

After making changes, always validate your configuration:

rsyslogd -N1
systemctl restart rsyslog
tail -f /var/log/syslog | grep "omfwd"

Check queue status with:

rsyslogd -o /var/lib/rsyslog/queue.stat -f /etc/rsyslog.conf

If logs aren't forwarding properly:

1. Verify network connectivity: telnet server1 514
2. Check queue directories: ls -l /var/spool/rsyslog/
3. Increase debug level: rsyslogd -dn
4. Verify disk space: df -h /var/spool/rsyslog