During routine network troubleshooting, we encountered a perplexing scenario where our Ubuntu 8.04 firewall would abruptly terminate HTTP connections by sending RST packets. The pattern emerged specifically when web servers sent multiple response packets before receiving client ACKs.
The Wireshark trace revealed this sequence:
1. Normal TCP handshake (SYN/SYN-ACK/ACK)
2. HTTP GET request and initial ACK
3. Server sends response in 8 packets including FIN/PSH
4. Firewall injects RST before client ACKs arrive
5. Client continues sending delayed ACKs into the voidThe firewall's routing configuration showed potential red flags:
Destination Gateway Genmask Flags Metric Ref Use Iface
10.1.1.0 0.0.0.0 255.255.255.240 U 0 0 0 eth1
192.168.126.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.60.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 10.1.1.15 0.0.0.0 UG 0 0 0 eth1Key findings from kernel logs:
[16406874.374588] raw pre IN=eth1 SRC=172.16.1.2 DST=10.1.1.1 PROTO=TCP SPT=80
[16406874.374625] mangle pre IN=eth1 ...
[16406874.374667] mangle in IN=eth1 ...
[16406874.374699] filter in IN=eth1 ...
[16406874.374780] mangle out OUT=eth1 SRC=10.1.1.1 DST=172.16.1.2 PROTO=TCP RSTThe problematic packet flow suggests a state tracking issue:
Working flow:
raw-pre -> conntrack -> mangle-pre -> nat-pre -> routing (forward)
-> mangle-fwd -> filter-fwd -> mangle-post -> nat-post
Failing flow:
raw-pre -> conntrack -> mangle-pre -> nat-pre -> routing (local)
-> mangle-input -> filter-input -> RST generationFor immediate mitigation:
# Increase TCP buffer sizes
sysctl -w net.ipv4.tcp_rmem="4096 87380 6291456"
sysctl -w net.ipv4.tcp_wmem="4096 16384 4194304"
# Adjust conntrack timeouts
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=432000
# Add explicit ACCEPT rules before potential DROPs
iptables -I FORWARD -p tcp --dport 80 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPTThe network topology reveals potential routing complexities:
+------------+ +----------+ +----------------+
| Firewall | | Router | | Client Network |
(Internet) ----| 10.1.1.1 |-------| |--------| 192.168.126.0/24|
| 192.168.60.254 | | | | |
+------------+ +----------+ +----------------+When TRACE target isn't available:
# Comprehensive logging rules
iptables -t raw -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "raw-pre "
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "mangle-pre "
iptables -t nat -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "nat-pre "
iptables -t mangle -A INPUT -p tcp --dport 80 -j LOG --log-prefix "mangle-in "
iptables -t filter -A INPUT -p tcp --dport 80 -j LOG --log-prefix "filter-in "Potential problematic modules to investigate:
lsmod | grep -E 'nf_conntrack|iptable|nf_nat'
modinfo nf_conntrack_ipv4
modinfo iptable_natAfter analyzing the packet capture, we can see the firewall unexpectedly sends a RST packet (sequence 2134) in response to the web server's FIN/ACK packet (2133). This occurs specifically when:
- The web server sends multiple response packets rapidly
- The client hasn't yet acknowledged all packets
- The final packet contains both FIN and PSH flags
The key finding from the iptables logging shows that packet 2133 gets routed differently:
[16406874.374699] filter in IN=eth1 OUT= MAC=... SRC=172.16.1.2 DST=10.1.1.1 ... ACK PSH FIN URGP=0Instead of being forwarded (like previous packets), this packet gets routed to local input chain. This suggests a conntrack state tracking issue in the ancient 2.6.24 kernel.
The asymmetric routing path complicates matters:
Web Server (172.16.1.2) → Firewall (10.1.1.1) → Router → Client (192.168.126.161)The firewall's routing table shows:
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.126.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0For Ubuntu 8.04's problematic conntrack implementation, try these fixes:
# Disable problematic TCP window tracking
echo 0 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
# Increase conntrack table size
echo 65536 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max
# Adjust TCP timeout settings
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=432000Add these rules before your main filter chain:
# Accept all FIN packets regardless of state
iptables -I INPUT -p tcp --tcp-flags FIN,ACK FIN,ACK -j ACCEPT
# Log problematic packets for debugging
iptables -N LOG_DROPPED
iptables -A LOG_DROPPED -j LOG --log-prefix "DROPPED: " --log-level 6
iptables -A LOG_DROPPED -j DROP
# Catch RST packets before they're sent
iptables -I OUTPUT -p tcp --tcp-flags RST RST -j LOG --log-prefix "RST_SENT: "Since Ubuntu 8.04 reached EOL in 2013, consider upgrading to a supported version. If stuck, this C code can help monitor the issue:
#include <linux/netfilter.h>
#include <libnetfilter_queue/libnetfilter_queue.h>
static int cb(struct nfq_q_handle *qh, struct nfgenmsg *nfmsg,
struct nfq_data *nfa, void *data) {
struct nfqnl_msg_packet_hdr *ph;
unsigned char *payload;
ph = nfq_get_msg_packet_hdr(nfa);
payload = nfq_get_payload(nfa, &ph);
// Check for FIN/ACK packets
if (ph->hw_protocol == htons(ETH_P_IP) &&
(payload[33] & (TCP_FLAG_FIN|TCP_FLAG_ACK))) {
printf("Detected FIN/ACK packet\n");
return nfq_set_verdict(qh, ntohl(ph->packet_id), NF_ACCEPT, 0, NULL);
}
return nfq_set_verdict(qh, ntohl(ph->packet_id), NF_ACCEPT, 0, NULL);
}
int main(int argc, char **argv) {
struct nfq_handle *h;
struct nfq_q_handle *qh;
h = nfq_open();
qh = nfq_create_queue(h, 0, &cb, NULL);
nfq_set_mode(qh, NFQNL_COPY_PACKET, 0xffff);
int fd = nfq_fd(h);
char buf[4096];
while (recv(fd, buf, sizeof(buf), 0) >= 0) {
nfq_handle_packet(h, buf, recv(fd, buf, sizeof(buf), 0));
}
}Use this command to catch the problematic sequence:
tcpdump -i eth1 'tcp[tcpflags] & (tcp-fin|tcp-ack) == (tcp-fin|tcp-ack)' \
and host 172.16.1.2 and port 80 -vv -n