Forwarding Custom Log Files (Outside /var/log) to Remote Syslog Server with Rsyslog


1 views

While rsyslog excels at handling system logs in /var/log, many applications write logs to custom locations like /www/myapp/log/test.log. Traditional syslog configurations don't automatically capture these files.

Here's how to set up forwarding for /www/myapp/log/test.log:

# Create a new rsyslog configuration file
sudo nano /etc/rsyslog.d/30-myapp.conf

Add the following configuration:

# Input module for monitoring the file
module(load="imfile" PollingInterval="10")

# Define the custom log file
input(
    type="imfile"
    File="/www/myapp/log/test.log"
    Tag="myapp"
    Severity="info"
    Facility="local7"
)

# Forward to remote server (replace with your server IP)
local7.* @@remote-syslog-server:514

imfile module: This built-in module allows rsyslog to monitor arbitrary files. The PollingInterval determines how often the file is checked for new entries.

Tag parameter: This identifier will appear in the remote logs, making it easier to filter messages from this specific application.

After saving the file, test your configuration:

sudo rsyslogd -N1

If no errors appear, restart rsyslog:

sudo systemctl restart rsyslog

For applications that rotate logs, add these parameters:

input(
    type="imfile"
    File="/www/myapp/log/test.log"
    Tag="myapp"
    Severity="info"
    Facility="local7"
    readTimeout="10"
    reopenOnTruncate="on"
)
  • Ensure the rsyslog user has read permissions for the log file
  • Check /var/log/syslog for rsyslog's own error messages
  • Verify the remote server is listening on UDP/TCP 514

When dealing with application logs stored outside the standard /var/log directory, traditional rsyslog configurations won't automatically capture these logs. A common scenario is web applications writing logs to directories like /www/myapp/log/ that need centralized logging.

First, we need to configure rsyslog to monitor our custom log file. Edit /etc/rsyslog.conf:


# Load imfile module for reading text files
module(load="imfile")

# Input definition for our custom log
input(type="imfile"
      File="/www/myapp/log/test.log"
      Tag="myapp:"
      Severity="info"
      Facility="local7")

Add the forwarding rule to send logs to your remote server (192.168.1.100 in this example):


# Define template for message format
template(name="RemoteFormat" type="string" string="<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%")

# Forward to remote server using RFC5424 format
local7.* action(type="omfwd" 
               target="192.168.1.100" 
               port="514"
               protocol="tcp"
               template="RemoteFormat"
               queue.size="100000"
               queue.type="LinkedList"
               action.resumeRetryCount="-1")

For production environments, consider these enhancements:


# Add file monitoring with inotify (better performance)
input(type="imfile"
      File="/www/myapp/log/test.log"
      Tag="myapp:"
      Severity="info"
      Facility="local7"
      PersistStateInterval="100"
      readTimeout="10"
      reopenOnTruncate="on")

# Enable disk-assisted queuing for network issues
action(type="omfwd"
       target="192.168.1.100"
       port="514"
       protocol="tcp"
       queue.filename="fwdqueue"
       queue.maxdiskspace="2g"
       queue.saveonshutdown="on"
       queue.type="LinkedList"
       action.resumeRetryCount="-1")

After making changes, test your configuration:


# Check syntax
sudo rsyslogd -N1

# Restart rsyslog
sudo systemctl restart rsyslog

# Follow system logs for errors
journalctl -u rsyslog -f

To test log forwarding, manually generate a test message:


logger -p local7.info -t myapp "Test message from custom log file"
  • Verify file permissions - rsyslog must have read access to the log file
  • Check SELinux contexts if running on RHEL/CentOS
  • Ensure proper network connectivity between hosts
  • Monitor queue statistics with rsyslogd -N1 -q