When working with IPSEC in tunnel mode, packets go through two distinct phases:

1. Encrypted phase: ESP/AH packets (protocols 50/51) arriving at your interface
2. Decrypted phase: The actual payload after IPSEC processing

To create firewall rules that distinguish between these phases, we need to understand how the Linux kernel processes IPSEC packets.

The key to solving this is iptables' policy match module, which checks the IPSEC policy applied to a packet. Here's the magic:

iptables -A INPUT -m policy --pol ipsec --dir in -p tcp --dport 8443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8443 -j DROP

These rules accomplish:

• First rule matches packets that went through IPSEC processing (--pol ipsec --dir in)
• Second rule drops all other traffic to port 8443

Here's a more complete example that includes state tracking and logging:

# Clear existing rules
iptables -F
iptables -X

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow established connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow localhost
iptables -A INPUT -i lo -j ACCEPT

# IPSEC-specific rules
iptables -A INPUT -m policy --pol ipsec --dir in -p tcp --dport 8443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8443 -j LOG --log-prefix "BLOCKED-NON-IPSEC: "
iptables -A INPUT -p tcp --dport 8443 -j DROP

# Optional: Allow IKE (UDP 500) and NAT-T (UDP 4500)
iptables -A INPUT -p udp --dport 500 -j ACCEPT
iptables -A INPUT -p udp --dport 4500 -j ACCEPT

To verify your setup:

# Check active policies
ip xfrm policy

# Monitor iptables hits
iptables -L -v -n

# Test connectivity
# From inside the tunnel:
curl https://your.server:8443

# From outside (should fail):
curl https://your.server:8443

Remember that IPSEC policies are processed before iptables rules, so the decrypted packets will have different characteristics than the original ESP packets.

• Ensure the xt_policy kernel module is loaded (lsmod | grep xt_policy)
• Check kernel support with grep XT_POLICY /boot/config-$(uname -r)
• If using NAT, ensure proper ip xfrm policies are in place
• For complex setups, consider using nftables instead, which has better IPSEC integration

When working with IPSEC in tunnel mode, packets undergo several transformations:

1. Encrypted ESP packets arrive at the interface (e.g., eth0)
2. IPSEC stack decrypts them
3. Decrypted packets enter the network stack with new headers

The key to filtering post-decryption traffic lies in iptables' policy match module. This matches packets based on their IPSEC policy:

sudo iptables -A INPUT -m policy --pol ipsec --dir in -p tcp --dport 8443 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 8443 -j DROP

Here's a full configuration example for securing port 3306 (MySQL) to only accept IPSEC-decrypted traffic:

# Load the policy match module
sudo modprobe xt_policy

# Allow IPSEC-encapsulated packets
sudo iptables -A INPUT -p esp -j ACCEPT
sudo iptables -A INPUT -p ah -j ACCEPT
sudo iptables -A INPUT -p udp --dport 500 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 4500 -j ACCEPT

# Allow decrypted traffic to MySQL
sudo iptables -A INPUT -m policy --pol ipsec --dir in -p tcp --dport 3306 -j ACCEPT

# Block all other MySQL access
sudo iptables -A INPUT -p tcp --dport 3306 -j DROP

Check that the rules are working as expected:

# List all iptables rules with line numbers
sudo iptables -L -n -v --line-numbers

# Test connectivity from inside and outside the tunnel
telnet your_server_ip 3306   # Should fail
ssh -L 3306:localhost:3306 tunnel_user@gateway   # Should succeed

For enhanced security, consider isolating the service in a network namespace:

# Create a new namespace
sudo ip netns add secure-ns

# Move the decrypted interface
sudo ip link set vti0 netns secure-ns

# Configure iptables in the namespace
sudo ip netns exec secure-ns iptables -A INPUT -m policy --pol ipsec --dir in -j ACCEPT
sudo ip netns exec secure-ns iptables -A INPUT -j DROP

If the rules aren't working:

1. Verify IPSEC is actually established: sudo ip xfrm state
2. Check kernel logs for policy errors: dmesg | grep policy
3. Ensure the xt_policy module is loaded: lsmod | grep policy
4. Verify the direction (--dir in/out) matches your traffic flow