Debugging Excessive DHCP Solicit Messages in EC2: Understanding XMT Log Entries and Network Interface Behavior


4 views

When examining /var/log/messages on AWS EC2 instances, the repetitive XMT: Solicit on eth0 entries indicate DHCPv6 solicitation messages being transmitted by the dhclient process. The 'XMT' prefix specifically denotes transmission (send) operations in DHCP client logs.

# Typical log sequence example
Jul 12 09:34:15 ip-172-31-16-25 dhclient[2187]: XMT: Solicit on eth0, interval 112321ms
Jul 12 09:35:47 ip-172-31-16-25 dhclient[2187]: XMT: Solicit on eth0, interval 111231ms

In AWS environments, this occurs because:

  • The instance attempts IPv6 address configuration through DHCPv6
  • EC2's metadata service (169.254.169.254) requests aren't properly terminating the DHCP process
  • Network interface reconfiguration triggers repeated solicitations

DHCPv6 clients send Solicit messages (equivalent to DHCPDISCOVER in IPv4) when:

// Simplified DHCPv6 state machine logic
if (no_IPv6_configuration && interface_up) {
    send_solicit();
    start_retransmission_timer();
    while (no_reply && retry_counter < MAX_RETRIES) {
        exponential_backoff();
        send_solicit();
    }
}

To prevent excessive logs in production environments:

# Option 1: Disable DHCPv6 in /etc/dhcp/dhclient6.conf
interface "eth0" {
    send dhcp6.client-id 00:01:00:01:27:71:ab:5d:aa:bb:cc:dd:ee:ff;
    request;
    script "/etc/dhcp/dhclient-script";
    ignore dhcp6.solicit;
}

# Option 2: Update network configuration
# For Amazon Linux 2/CentOS:
sudo sed -i '/DHCPV6C=/s/yes/no/' /etc/sysconfig/network-scripts/ifcfg-eth0
sudo systemctl restart network

Verify current DHCP behavior with:

# Check active DHCP processes
ps aux | grep dhclient

# Monitor real-time DHCP traffic
sudo tcpdump -i eth0 port 546 or port 547 -vv

# Check interface IPv6 configuration
ip -6 addr show dev eth0

While usually benign, frequent solicits may signal:

  • Network interface flaps (check ifup/ifdown events)
  • DHCP server unavailability
  • Incorrect MTU settings causing packet drops

For persistent cases, consider modifying the retransmission parameters in /etc/dhcp/dhclient.conf:

timeout 60;
retry 10;
select-timeout 5;
initial-interval 2;

The XMT: Solicit on eth0 messages in your EC2 instance logs indicate DHCPv6 client activity. Unlike DHCPv4 which uses DISCOVER-OFFER-REQUEST-ACK, DHCPv6 utilizes Solicit-Advertise-Request-Reply sequence.

EC2 instances leverage both the metadata service (169.254.169.254) and DHCP for network configuration. The logs show two parallel processes:

# Metadata service access
ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/...

# DHCPv6 client activity
dhclient[2187]: XMT: Solicit on eth0, interval 112321ms

Excessive Solicit messages typically indicate:

  • No DHCPv6 server responding on the network
  • Misconfigured IPv6 stack
  • Interface flapping triggering repeated requests
# Check DHCPv6 client status
sudo dhclient -6 -r eth0  # Release existing lease
sudo dhclient -6 -d -v eth0  # Run in foreground with debug

# Verify IPv6 configuration
ip -6 addr show dev eth0
ip -6 route show

# Check EC2 network interfaces
curl http://169.254.169.254/latest/meta-data/network/interfaces/macs/

For EC2 instances not needing IPv6, consider disabling DHCPv6:

# /etc/dhcp/dhclient.conf
interface "eth0" {
    ignore unknown-service dhcpv6;
    request subnet-mask, broadcast-address, time-offset, routers,
        domain-name, domain-name-servers, host-name;
}

Create a script to monitor DHCPv6 behavior:

#!/bin/bash
while true; do
    echo "$(date) - Checking DHCPv6 status"
    journalctl -u NetworkManager --since "1 minute ago" | grep -i dhcpv6
    ip -6 addr show dev eth0
    sleep 60
done