Deep Dive: Connection Tracking in iptables – State vs. Conntrack Modules and Performance Optimization


4 views

In modern Linux kernels, both -m state and -m conntrack ultimately use the same underlying netfilter connection tracking system (nf_conntrack). The key differences:

# Legacy state module (deprecated in newer kernels)
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Modern conntrack module (recommended)
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

While functionally equivalent for basic state matching, the conntrack module provides additional features:

  • Extended matching capabilities (--ctproto, --ctorigsrc, etc.)
  • Better integration with conntrack-tools
  • Future-proof implementation

Contrary to PF's keepstate approach, netfilter's connection tracking activates globally when:

  1. The nf_conntrack module loads
  2. Any rule references connection state (even just one rule)

This means all traffic flows are tracked, not just those matching specific rules. The tracking begins at the first packet of each flow.

Smart rule ordering significantly impacts performance. Place stateful rules early:

# Optimal ordering example
iptables -N STATE_TRACKING
iptables -A STATE_TRACKING -m conntrack --ctstate INVALID -j DROP
iptables -A STATE_TRACKING -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Jump to state tracking early
iptables -A FORWARD -j STATE_TRACKING

# Then process NEW connections
iptables -A FORWARD -p tcp --syn --dport 22 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -p tcp --syn --dport 80 -m conntrack --ctstate NEW -j ACCEPT

When the connection table fills (check with cat /proc/sys/net/netfilter/nf_conntrack_max):

  • New connections may be dropped even if rules would allow them
  • ESTABLISHED traffic continues flowing (tracking works for existing entries)
  • Stateless rules (without -m conntrack) continue functioning

Solution: Either:

  1. Increase nf_conntrack_max (with corresponding nf_conntrack_buckets)
  2. Implement a fallback stateless rule (not recommended for security)
  3. Use connection tracking more selectively (complex to implement)
#!/bin/sh
# Conntrack-aware firewall for Linux gateway

# Set sane defaults
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Connection tracking optimization
iptables -N CT_STATES
iptables -A CT_STATES -m conntrack --ctstate INVALID -j DROP
iptables -A CT_STATES -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Fast paths for tracked connections
iptables -A FORWARD -j CT_STATES
iptables -A INPUT -j CT_STATES

# New connections
iptables -A FORWARD -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT

# ICMP handling
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A FORWARD -p icmp --icmp-type echo-request -j ACCEPT

Remember to monitor conntrack table usage with conntrack -L or cat /proc/net/nf_conntrack.


In modern Linux kernels (2.6.20+), these two rules are functionally equivalent:

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

The state module is actually just a legacy wrapper around the newer conntrack infrastructure. Both trigger loading of the nf_conntrack kernel module when first invoked.

When using conntrackd for state synchronization between firewalls, you should use the explicit -m conntrack syntax for consistency. While both forms work, the conntrack version makes the dependency clearer in your ruleset.

Unlike BSD's PF where you explicitly enable state tracking per rule with keepstate, Netfilter's connection tracking activates globally when:

  1. The nf_conntrack module loads (automatically via iptables or manually)
  2. The first packet enters the network stack

This means all traffic flows are tracked regardless of whether your rules reference connection state. The tracking occurs before iptables rules evaluation.

The most efficient approach places ESTABLISHED/RELATED rules early:

# Efficient ordering example
iptables -P FORWARD DROP
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -p tcp --dport 22 --syn -m conntrack --ctstate NEW -j ACCEPT

This structure ensures:

  • Invalid packets are dropped immediately
  • ESTABLISHED traffic bypasses further rule evaluation
  • Only NEW connections undergo full ruleset inspection

When the connection tracking table fills:

# Check current table size
cat /proc/sys/net/netfilter/nf_conntrack_max

# Monitor usage
conntrack -L | wc -l

Critical considerations:

  • Rules without state matching continue working (pure port-based rules)
  • INVALID drops become dangerous - may block legitimate traffic during congestion
  • Solutions:
    • Increase nf_conntrack_max
    • Reduce nf_conntrack_tcp_timeout_* values
    • Implement connection rate limiting

Here's a robust FORWARD chain configuration for a gateway:

#!/bin/sh
# Connection tracking parameters
sysctl -w net.netfilter.nf_conntrack_max=300000
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=5400

# Basic policy
iptables -P FORWARD DROP

# Connection tracking rules
iptables -A FORWARD -m conntrack --ctstate INVALID -j LOG --log-prefix "INVALID: "
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# New connections
iptables -A FORWARD -p tcp --syn -m conntrack --ctstate NEW -m multiport --dports 80,443 -j ACCEPT
iptables -A FORWARD -p udp -m conntrack --ctstate NEW -m multiport --dports 53,123 -j ACCEPT

# Rate limiting for DoS protection
iptables -A FORWARD -p tcp --syn -m conntrack --ctstate NEW -m limit --limit 60/minute --limit-burst 30 -j ACCEPT
iptables -A FORWARD -p tcp --syn -m conntrack --ctstate NEW -j DROP