When running a server cluster behind a load balancer, health check probes (typically every 5-10 seconds) can flood your syslog with repetitive entries. These entries provide no real value for troubleshooting but consume storage and make log analysis harder.

rsyslog offers several powerful filtering approaches:

# Basic property-based filter
if $msg contains "HEALTHCHECK" then stop

# Regex example
if $msg =~ "/healthcheck.*200 OK/" then stop

# Hostname matching
if $fromhost-ip == "192.168.1.100" and $msg contains "GET /health" then stop

For NGINX access logs as an example:

# /etc/rsyslog.d/01-filter.conf
# Drop health check entries
if $programname == 'nginx' and $msg contains '/healthcheck' then {
    action(type="omfile" file="/var/log/nginx/healthchecks.log")
    stop
}

For complex environments:

# Rate-limiting approach
if $msg contains "ELB-HealthChecker" then {
    action(type="omfile" file="/var/log/elb_health.log")
    # Limit to 1 entry per minute
    action(type="omfwd" Target="remote.example.com" Port="514"
           Protocol="tcp" queue.filename="elb_fwd"
           queue.maxdiskspace="100m"
           queue.size="10000"
           queue.type="LinkedList"
           action.resumeRetryCount="-1"
           queue.saveonshutdown="on")
    stop
}

Always test changes with:

rsyslogd -N1  # Validate config syntax
systemctl restart rsyslog
tail -f /var/log/syslog | grep -v healthcheck  # Verify filtering

When running a Linux server cluster behind a load balancer, you'll likely encounter excessive log entries from frequent health checks. These 5-second probes create significant log noise while providing minimal operational value. Here's a typical unwanted entry:

Apr 12 09:15:23 webserver01 kernel: [UFW BLOCK] IN=eth0 OUT= MAC=01:23:45:67:89:ab SRC=10.0.0.100 DST=10.0.1.10 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=1234 PROTO=TCP SPT=12345 DPT=80 WINDOW=0 RES=0x00 SYN URGP=0 

1. Property-Based Filtering

The most efficient method uses rsyslog's property-based filters to drop messages matching specific patterns:

# /etc/rsyslog.d/01-filter-healthchecks.conf
:msg, contains, "SRC=10.0.0.100 DPT=80" ~
:msg, contains, "UFW BLOCK" and $msg contains "DPT=80" ~

The tilde (~) at the end means "discard this message". The second example shows combining multiple conditions.

2. Using rsyslog Templates

For more complex filtering, create a template with custom logic:

template(name="healthcheck_filter" type="list") {
    property(name="msg")
    if $msg contains "SRC=10.0.0.100" and $msg contains "DPT=80" then {
        stop # Discard message
    }
}

If you want to keep some health check logs but reduce volume:

# Limit to 1 message per minute for health checks
$SystemLogRateLimitInterval 60
$SystemLogRateLimitBurst 1

if $msg contains "HealthCheck" then {
    action(type="omfile" file="/var/log/healthchecks.log")
    stop
}

After making changes, verify your configuration:

rsyslogd -N1 # Test config syntax
systemctl restart rsyslog
tail -f /var/log/syslog | grep --line-buffered "UFW BLOCK"

For persistent monitoring, consider adding this to your logrotate configuration to ensure filters remain effective after log rotation.