During our investigation of TCP exhaustion in our server application, we encountered a perplexing scenario where netstat -b showed established connections to clients that were physically powered off. This contradicts fundamental TCP/IP principles, yet the evidence suggests otherwise.

Technically, a TCP connection cannot remain truly "established" when:

• The client's NIC is disabled
• The client system is powered off
• The client's TCP stack is terminated

However, several conditions can create the illusion of persistent connections:

// Example of server-side connection tracking
struct tcp_connection {
    int fd;
    time_t last_activity;
    enum { ACTIVE, CLOSE_WAIT, TIME_WAIT } state;
    // Missing proper timeout handling
};

Network-level explanations:

• Intermediate devices (NAT, firewalls) keeping mappings alive
• TCP keepalive misconfiguration (default 2+ hours in many OS)
• Unacknowledged FIN packets due to network issues

Use these Linux commands to investigate:

# Check connection states with timers
ss -t -o state established

# Monitor TCP retransmissions
nstat -az TcpRetransSegs TcpEstabResets

# Check kernel TCP memory
sysctl net.ipv4.tcp_mem

Server-side hardening measures:

// Recommended sysctl settings (Linux)
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
echo 60 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
echo 1 > /proc/sys/net/ipv4/tcp_keepalive_enabled

Application-level improvements:

// Python example using socket timeouts
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(30.0)  # 30-second operational timeout
s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

We discovered our server's internal connection table wasn't properly synchronized with the OS TCP stack. The fix involved:

// Before (buggy):
void handle_disconnect(int sockfd) {
    remove_from_connection_table(sockfd); 
    // Missing actual close() call
}

// After (fixed):
void handle_disconnect(int sockfd) {
    shutdown(sockfd, SHUT_RDWR);
    close(sockfd);
    remove_from_connection_table(sockfd);
}

Implement these proactive measures:

• Connection aging scripts using ss command
• TCP metrics monitoring (retransmits, orphaned sockets)
• Regular connection table audits

---

In modern networked applications, we occasionally encounter situations where the server reports an "ESTABLISHED" TCP connection while the client has already terminated. This creates TCP exhaustion problems, particularly in long-running server applications handling thousands of connections.

A proper TCP connection termination follows the four-way handshake:

Client -> FIN -> Server
Client <- ACK <- Server
Client <- FIN <- Server
Client -> ACK -> Server

However, when clients disconnect abruptly (power loss, network failure, or crash), this graceful termination sequence doesn't occur.

Several technical reasons can cause this discrepancy:

• TCP Keepalive: Disabled by default on many systems (2 hours idle time before probing)
• Network Partition: Client becomes unreachable but didn't send FIN
• OS Differences: Varying TCP stack implementations handle timeouts differently
• Application Bugs: Client application crashes without closing sockets

Here's how to identify stale connections programmatically (Linux example):

# Check TCP connection states
netstat -tnop | grep ESTABLISHED

# Monitor keepalive probes
tcpdump -i eth0 'tcp[tcpflags] & (tcp-ack) != 0 and \ 
  (tcp[tcpflags] & (tcp-syn) == 0) and (tcp[tcpflags] & (tcp-fin) == 0)'

Implement these measures server-side to prevent connection exhaustion:

1. TCP Keepalive Configuration

// Python example
import socket

def set_keepalive(sock, after_idle_sec=60, interval_sec=10, max_fails=5):
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, after_idle_sec)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, interval_sec)
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, max_fails)

2. Application-Level Timeouts

// Node.js example with connection timeout
const net = require('net');

const server = net.createServer((socket) => {
  socket.setTimeout(30000); // 30 seconds
  socket.on('timeout', () => socket.end());
});

3. OS-Level Tuning

Linux kernel parameters for faster dead connection detection:

# /etc/sysctl.conf
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 6
net.ipv4.tcp_fin_timeout = 30

Implement these checks in your monitoring system:

# Prometheus query for stale connections
sum by (instance) (tcp_connection_states{state="ESTABLISHED"}) - 
sum by (instance) (tcp_connection_states{state="ESTABLISHED"} offset 5m) > 0