AWS VPC NAT Instance: Troubleshooting iptables Port Forwarding for SSH Access to Private Subnet


4 views

In this AWS VPC setup, we have a NAT instance (10.0.0.54) acting as gateway between public and private subnets. Server A (10.0.1.243) resides in the private subnet and needs to be accessible via SSH from the internet through port forwarding.

The existing iptables rules on the NAT instance include:

# PREROUTING rule for port forwarding
iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.0.1.243:22

# MASQUERADE rule for outbound traffic
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 10.0.0.0/16

While tcpdump on the NAT instance shows packets being forwarded:

09:59:13.738316 IP [Public_IP].51709 > 10.0.1.243.ssh: SYN packet

No corresponding packets appear on Server A's interface, suggesting the forwarded packets aren't reaching their destination.

The issue likely stems from missing FORWARD chain rules. Try adding these essential iptables rules:

# Allow forwarded packets to Server A
iptables -A FORWARD -p tcp -d 10.0.1.243 --dport 22 -j ACCEPT

# Accept related/established connections
iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

Here's the full set of commands needed for proper port forwarding:

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward
sysctl -p

# DNAT rule for incoming connections
iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.0.1.243:22

# SNAT/MASQUERADE rule
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -s 10.0.1.243

# FORWARD chain rules
iptables -A FORWARD -p tcp -d 10.0.1.243 --dport 22 -j ACCEPT
iptables -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT

After implementing these rules, verify with:

# Check NAT rules
iptables -t nat -L -n -v

# Verify connection from NAT to Server A
nc -zv 10.0.1.243 22

# Test end-to-end connectivity
ssh -p 2222 [NAT_Public_IP]

Ensure these additional configurations are correct:

  • NAT instance's security group must allow inbound TCP/2222
  • Server A's security group must allow TCP/22 from NAT's private IP (10.0.0.54)
  • Route tables must direct private subnet traffic through the NAT instance
  • Disable source/destination checks on the NAT instance

For production environments, consider using AWS Session Manager for SSH access without port forwarding:

# Install SSM Agent on Server A
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

# Connect via AWS CLI
aws ssm start-session --target i-[instance-id]

If issues persist:

  1. Verify EC2 instance roles and permissions
  2. Check VPC flow logs for packet drops
  3. Test with temporary permissive security groups
  4. Inspect kernel logs on both NAT and Server A
  5. Confirm the NAT instance is properly sized (t2.small minimum recommended)

Here's a common AWS networking challenge I recently faced while configuring a production environment:

Internet → NAT Instance (Public Subnet) → Server A (Private Subnet)

On the NAT instance (10.0.0.54), I've implemented these critical components:

# Enabled IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# NAT rule for port forwarding
iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.0.1.243:22

# Masquerade rule
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Packet tracing revealed an interesting pattern:

# On NAT instance (shows packets arriving):
tcpdump -n -i eth0 dst 10.0.1.243 and port 22
09:59:13.738316 IP client.ip.51709 > 10.0.1.243.ssh: [SYN] Seq=1868541786

# On Server A (shows NO packets from NAT):
tcpdump -n src 10.0.0.54
[empty output]

After extensive testing, I discovered three critical missing pieces:

# 1. Missing FORWARD chain rule (critical for VPC environments)
iptables -A FORWARD -i eth0 -o eth0 -d 10.0.1.243 -p tcp --dport 22 -j ACCEPT

# 2. Missing conntrack module support
modprobe nf_conntrack_ipv4
echo "options nf_conntrack hashsize=131072" > /etc/modprobe.d/nf_conntrack.conf

# 3. Missing source-based routing return path
iptables -t nat -A POSTROUTING -s 10.0.1.243 -j SNAT --to-source 10.0.0.54

Here's the full working configuration for AWS VPC port forwarding:

#!/bin/bash

# Enable IP forwarding
sysctl -w net.ipv4.ip_forward=1

# Load necessary modules
modprobe nf_conntrack
modprobe nf_conntrack_ipv4

# Flush existing rules
iptables -F
iptables -t nat -F

# Set default policies
iptables -P FORWARD DROP

# Allow established connections
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT

# Specific forwarding rule
iptables -A FORWARD -i eth0 -o eth0 -d 10.0.1.243 -p tcp --dport 22 -m state --state NEW -j ACCEPT

# NAT rules
iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.0.1.243:22
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.0.1.243 -j SNAT --to-source 10.0.0.54

Remember these VPC-specific requirements:

  • Source/dest check must be disabled on NAT instance
  • Security groups must allow inbound 2222 to NAT and outbound 22 to Server A
  • Route tables must point private subnets to NAT instance