Linux Kernel “net_ratelimit: callbacks suppressed” Error: Diagnosis and Tuning for Snort Performance


3 views

When working with network-intensive applications like Snort on Linux routers, you might encounter two related symptoms:

1. "snort packet recv contents failure: No buffer space available"
2. "net_ratelimit: 44 callbacks suppressed"

The first indicates packet buffer exhaustion, while the second reveals a rate-limiting mechanism kicking in.

The log shows multiple "martian source" entries, which are packets with impossible source addresses (like 216.59.11.21 claiming to come from 127.0.0.1). These trigger kernel callbacks for logging:

Jun  4 14:17:36 ilium kernel: [25743.259955] martian source 216.59.11.21 from 127.0.0.1
Jun  4 14:17:36 ilium kernel: [25743.259956] ll header: 00:30:48:7c:f8:10:00:24:c4:49:8d:00:08:00

Linux implements net_ratelimit() to prevent log flooding. When callbacks exceed:

  • Default: 10 messages every 5 seconds (configurable via /proc/sys/net/core/message_burst and message_cost)
  • Suppressed messages are counted and reported in batches

Your buffer adjustments were correct, but need complementary changes:

# Network stack tuning (add to /etc/sysctl.conf)
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_wmem = 1048576 4194304 16777216
net.ipv4.tcp_rmem = 1048576 4194304 16777216
net.core.netdev_max_backlog = 30000

# Rate limiting adjustments
net.core.message_burst = 100  # Allow more messages before throttling
net.core.message_cost = 5     # Lower cost per message

# Martian packet handling
net.ipv4.conf.all.log_martians = 0  # Disable if martian logs aren't critical
net.ipv4.conf.default.rp_filter = 2 # Strict reverse path filtering

Add these to your snort.conf:

config daq_var: buffer_size_mb=16
config pkt_count: 100000
config disable_engine: no

After changes, monitor with:

watch -n 1 "cat /proc/net/softnet_stat | awk '{print \$1,\$2,\$3}'"
journalctl -f -k -p warning

The first column in softnet_stat shows dropped packets - aim to keep it at 0.


When working with high-traffic network applications like Snort on Linux, you might encounter this kernel message:

net_ratelimit: 44 callbacks suppressed

This indicates the kernel's rate-limiting mechanism is actively suppressing network-related log messages. The number (44 in this case) shows how many identical messages were blocked during the rate-limiting window.

The message often appears alongside other network performance problems. In your case, you initially saw:

snort packet recv contents failure: No buffer space available

After increasing TCP buffers to 16MB using these sysctl settings:

#!/bin/sh -e
sysctl -w net.core.rmem_default='16777216'
sysctl -w net.core.wmem_default='16777216'
sysctl -w net.core.rmem_max='16777216'
sysctl -w net.core.wmem_max='16777216'
sysctl -w net.ipv4.tcp_wmem='1048576 4194304 16777216'
sysctl -w net.ipv4.tcp_rmem='1048576 4194304 16777216'
sysctl -w net.core.netdev_max_backlog='30000'

The log shows "martian source" messages appearing alongside the rate-limited warnings:

martian source 216.59.11.21 from 127.0.0.1, on dev eth0
ll header: 00:30:48:7c:f8:10:00:24:c4:49:8d:00:08:00

Martian packets are packets with impossible source addresses (like 127.0.0.1 coming from an external interface). These can trigger numerous kernel warnings that get rate-limited.

For Snort performance tuning, consider these additional adjustments:

# Increase socket buffer sizes
sysctl -w net.ipv4.tcp_mem='16777216 16777216 16777216'

# Disable martian packet logging (if acceptable)
sysctl -w net.ipv4.conf.all.log_martians=0
sysctl -w net.ipv4.conf.default.log_martians=0

# Adjust rate limiting thresholds
sysctl -w net.core.message_cost=50
sysctl -w net.core.message_burst=100

After making changes, monitor system behavior with:

# Watch kernel messages in real-time
dmesg -wT

# Check current network settings
sysctl -a | grep -E 'mem|backlog|ratelimit'

Remember that rate-limiting messages are often symptoms rather than root causes. The suppressed callbacks typically indicate an underlying network configuration or performance issue that needs addressing.