How to Monitor and Troubleshoot TCP Backlog Queue Usage Against net.core.somaxconn Limit


14 views

The net.core.somaxconn parameter defines the maximum number of pending connections that can be queued for accepting in the TCP/IP stack. When a server receives connection requests faster than the application can accept them, these connections enter a backlog queue controlled by this limit.

Several tools can help monitor how close you're getting to the somaxconn limit:

# Check current somaxconn value
sysctl net.core.somaxconn

# Monitor TCP connection states (run continuously)
watch -n 1 'ss -nt state syn-recv | wc -l'

# Alternative using netstat
netstat -tan | grep -c SYN_RECV

Here's a Python script to monitor backlog usage in real-time:

import time
import subprocess

def get_backlog_count():
    result = subprocess.run(['ss', '-nt', 'state', 'syn-recv'], 
                          stdout=subprocess.PIPE)
    return len(result.stdout.decode().splitlines()) - 1  # subtract header

def main():
    while True:
        current = get_backlog_count()
        max_conn = int(subprocess.run(['sysctl', '-n', 'net.core.somaxconn'],
                                    stdout=subprocess.PIPE).stdout)
        print(f"Backlog usage: {current}/{max_conn} ({current/max_conn:.1%})")
        time.sleep(1)

if __name__ == "__main__":
    main()

When your SYN_RECV count consistently approaches your somaxconn value:

  • Clients may experience connection timeouts
  • You'll see increased TCP retransmissions
  • SYN cookies might activate (check /proc/sys/net/ipv4/tcp_syncookies)

For production systems handling heavy connection loads:

# Set higher somaxconn value (persistent in /etc/sysctl.conf)
echo "net.core.somaxconn = 1024" >> /etc/sysctl.conf
sysctl -p

# Also adjust the application's listen() backlog parameter
# Example in Python:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.listen(1024)  # Should match or be slightly higher than somaxconn

Use these kernel parameters for deeper analysis:

# Enable TCP tracing
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_abort_on_overflow

# Check dropped connections
grep -i "tcp" /proc/net/netstat | grep -i "listen"

The net.core.somaxconn parameter defines the maximum number of pending connections that can be queued for a listening socket in Linux. When a TCP SYN packet arrives and the server is busy, these connections go into a backlog queue instead of being immediately rejected.

Check current connection states with:

ss -lnt | grep -E 'State|:'

Example output showing queue usage:

State      Recv-Q Send-Q Local Address:Port  Peer Address:Port
LISTEN     12     128    *:8080             *:*

Create a monitoring script (somaxconn_monitor.sh):

#!/bin/bash
PORT=8080
CURRENT=$(ss -lnt "sport = :$PORT" | awk 'NR==2 {print $2}')
MAX=$(sysctl -n net.core.somaxconn)
PERCENT=$((100*$CURRENT/$MAX))

echo "Current backlog: $CURRENT/$MAX ($PERCENT%)"
if [ $PERCENT -gt 80 ]; then
    echo "WARNING: Approaching somaxconn limit!"
fi

For deeper inspection using kernel parameters:

# View overflow statistics
grep -E 'TCP|overflow' /proc/net/netstat

Consider raising the limit when:

  • Recv-Q regularly reaches 70-80% of somaxconn
  • You observe SYN cookies in netstat output
  • Connection timeouts occur during traffic spikes