How to Monitor and Troubleshoot TCP Backlog Queue Usage Against net.core.somaxconn Limit


1 views

The net.core.somaxconn parameter defines the maximum number of pending connections that can be queued for accepting in the TCP/IP stack. When a server receives connection requests faster than the application can accept them, these connections enter a backlog queue controlled by this limit.

Several tools can help monitor how close you're getting to the somaxconn limit:

# Check current somaxconn value
sysctl net.core.somaxconn

# Monitor TCP connection states (run continuously)
watch -n 1 'ss -nt state syn-recv | wc -l'

# Alternative using netstat
netstat -tan | grep -c SYN_RECV

Here's a Python script to monitor backlog usage in real-time:

import time
import subprocess

def get_backlog_count():
    result = subprocess.run(['ss', '-nt', 'state', 'syn-recv'], 
                          stdout=subprocess.PIPE)
    return len(result.stdout.decode().splitlines()) - 1  # subtract header

def main():
    while True:
        current = get_backlog_count()
        max_conn = int(subprocess.run(['sysctl', '-n', 'net.core.somaxconn'],
                                    stdout=subprocess.PIPE).stdout)
        print(f"Backlog usage: {current}/{max_conn} ({current/max_conn:.1%})")
        time.sleep(1)

if __name__ == "__main__":
    main()

When your SYN_RECV count consistently approaches your somaxconn value:

  • Clients may experience connection timeouts
  • You'll see increased TCP retransmissions
  • SYN cookies might activate (check /proc/sys/net/ipv4/tcp_syncookies)

For production systems handling heavy connection loads:

# Set higher somaxconn value (persistent in /etc/sysctl.conf)
echo "net.core.somaxconn = 1024" >> /etc/sysctl.conf
sysctl -p

# Also adjust the application's listen() backlog parameter
# Example in Python:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.listen(1024)  # Should match or be slightly higher than somaxconn

Use these kernel parameters for deeper analysis:

# Enable TCP tracing
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > /proc/sys/net/ipv4/tcp_abort_on_overflow

# Check dropped connections
grep -i "tcp" /proc/net/netstat | grep -i "listen"

The net.core.somaxconn parameter defines the maximum number of pending connections that can be queued for a listening socket in Linux. When a TCP SYN packet arrives and the server is busy, these connections go into a backlog queue instead of being immediately rejected.

Check current connection states with:

ss -lnt | grep -E 'State|:'

Example output showing queue usage:

State      Recv-Q Send-Q Local Address:Port  Peer Address:Port
LISTEN     12     128    *:8080             *:*

Create a monitoring script (somaxconn_monitor.sh):

#!/bin/bash
PORT=8080
CURRENT=$(ss -lnt "sport = :$PORT" | awk 'NR==2 {print $2}')
MAX=$(sysctl -n net.core.somaxconn)
PERCENT=$((100*$CURRENT/$MAX))

echo "Current backlog: $CURRENT/$MAX ($PERCENT%)"
if [ $PERCENT -gt 80 ]; then
    echo "WARNING: Approaching somaxconn limit!"
fi

For deeper inspection using kernel parameters:

# View overflow statistics
grep -E 'TCP|overflow' /proc/net/netstat

Consider raising the limit when:

  • Recv-Q regularly reaches 70-80% of somaxconn
  • You observe SYN cookies in netstat output
  • Connection timeouts occur during traffic spikes