Debugging UDP DNS Resolution Failures in Docker Containers on RHEL 7 Hosts


2 views

While working with Docker containers on a RHEL 7.2 host (Docker 1.12.6), I encountered a perplexing networking issue where DNS resolution would fail inside containers despite working perfectly on the host. The error messages pointed to UDP packet loss when containers attempted to query DNS servers.

# Sample error from docker logs
Aug 08 11:44:27 myhost.example.com dockerd-current[17341]: 
time="2017-08-08T11:44:27.431772666+10:00" level=debug 
msg="Read from DNS server failed, read udp 172.18.0.6:38189->10.162.182.101:53: i/o timeout"

The immediate workaround was to force TCP DNS resolution by adding --dns-opt=use-vc to Docker's options:

# /etc/sysconfig/docker modification
OPTIONS="--dns=10.0.0.10 --dns=10.0.0.11 --dns-search=example.com --dns-opt=use-vc"

This worked because TCP's reliability mechanisms overcame whatever was causing UDP packets to disappear. However, this wasn't an ideal solution as it increased DNS resolution overhead.

The host had two bridges configured:

# Network bridge configuration
4: docker0:  mtu 1500
3: br-1d6a05c10468:  mtu 1500

Key iptables rules showed normal Docker isolation chains:

# iptables -n -L DOCKER-ISOLATION -v --line-numbers
Chain DOCKER-ISOLATION (1 references)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 DROP       all  --  br-1d6a05c10468 docker0  0.0.0.0/0            0.0.0.0/0           
2        0     0 DROP       all  --  docker0 br-1d6a05c10468  0.0.0.0/0            0.0.0.0/0

Docker logs revealed frequent Firewalld status checks:

Aug 04 17:33:34 myhost.example.com dockerd-current[2131]: 
time="2017-08-04T17:33:34.903371015+10:00" level=info 
msg="Firewalld running: true"

This turned out to be the root cause - Firewalld was silently dropping UDP packets between containers and the host. The solution was to either:

  1. Completely disable Firewalld (not recommended for production)
  2. Add explicit rules for Docker networks

Here's the proper way to configure Firewalld for Docker:

# Add Docker zones to Firewalld
firewall-cmd --permanent --new-zone=docker
firewall-cmd --permanent --zone=docker --add-source=172.17.0.0/16
firewall-cmd --permanent --zone=docker --add-source=172.18.0.0/16
firewall-cmd --permanent --zone=docker --add-port=53/udp
firewall-cmd --permanent --zone=docker --add-port=53/tcp
firewall-cmd --reload

This ensures UDP DNS packets can flow freely between containers and the host while maintaining other security boundaries.

To confirm the solution works:

# From inside a container
docker run --rm -it centos:7 bash
# ping 8.8.8.8  # Should work
# nslookup google.com  # Should now resolve

Additionally, packet capture confirms UDP flows:

# On the host
tcpdump -i any -n udp port 53

When running Docker containers on RHEL 7.2 with Docker 1.12.6, we encountered a peculiar issue where DNS resolution worked perfectly on the host but consistently failed inside containers. The symptoms pointed to UDP packet forwarding problems:

Aug 08 11:44:23 myhost.example.com dockerd-current[17341]: level=debug msg="Read from DNS server failed, read udp 172.18.0.6:38189->10.162.182.101:53: i/o timeout"

The host was configured with custom DNS settings in /etc/sysconfig/docker:

OPTIONS='--dns=10.0.0.10 --dns=10.0.0.11 --dns-search=example.com'

Both container and host shared identical /etc/resolv.conf:

search example.com
nameserver 10.0.0.10
nameserver 10.0.0.11

Forcing DNS over TCP worked immediately by adding:

OPTIONS="--dns-opt=use-vc"  # Forces TCP instead of UDP

This suggested our core issue was specifically with UDP packet forwarding between containers and the host.

The logs showed repeated Firewalld activity during Docker startup:

Aug 04 17:33:34 myhost.example.com dockerd-current[2131]: level=info msg="Firewalld running: true"

Despite RHEL 7 using Firewalld by default, Docker's iptables rules weren't properly coordinating with it. Here's the critical fix:

# Either completely disable Firewalld (not recommended)
systemctl disable firewalld --now

# OR configure proper integration
firewall-cmd --permanent --zone=trusted --add-interface=docker0
firewall-cmd --permanent --zone=trusted --add-interface=br-*
firewall-cmd --permanent --zone=trusted --add-port=53/udp
firewall-cmd --reload

For enterprise environments where Firewalld must remain active, implement this comprehensive solution:

# 1. Configure Docker DNS with TCP fallback
cat > /etc/sysconfig/docker <<EOF
OPTIONS='--dns=10.0.0.10 --dns=10.0.0.11 --dns-search=example.com --dns-opt=use-vc'
EOF

# 2. Configure Firewalld for Docker networks
firewall-cmd --permanent --new-zone=docker
firewall-cmd --permanent --zone=docker --add-source=172.17.0.0/16
firewall-cmd --permanent --zone=docker --add-source=172.18.0.0/16
firewall-cmd --permanent --zone=docker --add-port=53/udp
firewall-cmd --permanent --zone=docker --add-port=53/tcp
firewall-cmd --reload

# 3. Verify module loading
modprobe br_netfilter
echo "br_netfilter" > /etc/modules-load.d/br_netfilter.conf

# 4. Restart Docker
systemctl restart docker

Run these diagnostic commands to confirm resolution:

# From inside container:
docker run --rm -it rhel:7.3 bash -c "ping -c 1 10.0.0.10 && \
dig +tcp @10.0.0.10 google.com && \
dig +notcp @10.0.0.10 google.com"

# Check Firewalld rules
firewall-cmd --list-all --zone=docker
iptables -L -n -v | grep -i docker

The interaction between three components caused this failure:

  • Firewalld's default zone blocking container-originated UDP packets
  • Docker's bridge networking not properly whitelisted in firewalld
  • DNS resolver's default UDP preference with no automatic TCP fallback

The solution addresses all three aspects while maintaining security and functionality.