How to Filter Out Specific Kernel Messages in Rsyslog: Blocking “TCP Peer Unexpectedly Shrunk Window” Errors


3 views

When working with Linux systems, particularly those handling heavy network traffic, you might encounter repetitive kernel messages like:

[168707.740364] TCP: Peer 192.168.100.1:46199/41503 unexpectedly shrunk window 2027330493:2027331431 (repaired)

These messages can clutter your logs without providing actionable information. While your initial attempt with if $msg contains 'unexpectedly' then /dev/null seemed logical, rsyslog's filtering requires more precision when dealing with kernel messages.

Kernel messages first go through klogd before reaching rsyslog. The $msg property in your filter might not capture the full message as it appears in the syslog. We need to address both the kernel message format and rsyslog's processing pipeline.

Create a dedicated rsyslog filter file (recommended for maintainability):

# /etc/rsyslog.d/01-filter-kernel-msgs.conf
# Stop logging TCP window shrink messages
:msg, contains, "TCP: Peer" and contains, "unexpectedly shrunk window" ~
# Alternative regex version for more precise matching
:msg, regex, "TCP: Peer.*unexpectedly shrunk window" ~

The tilde (~) at the end tells rsyslog to discard matching messages. After adding this, restart rsyslog:

sudo systemctl restart rsyslog

To test if your filter works before applying it system-wide:

# Test configuration syntax
sudo rsyslogd -N1
# Run in foreground with debug output
sudo rsyslogd -dn | grep "TCP.*shrunk"

For persistent messages, consider adjusting kernel logging levels:

# Add to /etc/sysctl.conf
kernel.printk = 3 4 1 3
# Or block specific message types
echo 'kernel.printk = 3 4 1 3' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

This reduces the verbosity of kernel messages at the source.

  • Always document why you're filtering specific messages
  • Consider logging filtered messages to a separate file instead of discarding completely
  • Test filtering rules in a staging environment first
  • Remember that some network monitoring tools might need these messages

Many sysadmins encounter these annoying TCP window-related messages flooding their syslog:

[168707.740364] TCP: Peer 192.168.100.1:46199/41503 unexpectedly shrunk window 2027330493:2027331431 (repaired)

The common approach of using basic message filtering often doesn't work because:

  • The kernel messages appear before rsyslog processing
  • The complete message structure isn't what rsyslog sees

Option 1: Kernel-Level Filtering

Add this to /etc/rsyslog.conf before any rules:

# Kernel message filtering
:msg, contains, "TCP: Peer" and contains, "unexpectedly shrunk window" ~
:msg, contains, "TCP: Peer" and contains, "shrunk window" ~

Option 2: Using rsyslog Property Filters

More precise filtering using message properties:

if ($syslogtag == 'kernel:' and $msg contains 'shrunk window') then {
    action(type="omfile" file="/dev/null")
    stop
}

Option 3: Combining Multiple Conditions

For comprehensive filtering across different message formats:

if (
    ($msg contains 'unexpectedly shrunk window') or
    ($msg contains 'shrunk window') or
    ($msg contains 'window repair')
) then {
    /dev/null
    stop
}

After making changes:

sudo rsyslogd -N1
sudo systemctl restart rsyslog
  • Check /var/log/syslog for rsyslog's own errors
  • Use logger command to test filters
  • Consider adjusting kernel log level if messages persist

For chronic TCP window issues, investigate:

  • Network MTU settings
  • TCP stack parameters (/proc/sys/net/ipv4/)
  • Firewall or middlebox interference