Troubleshooting IPMI Remote Connection Failures: When ARP Works But Ping/SSH Fails


2 views

When your IPMI/BMC responds to ARP requests but refuses to answer pings or remote connections, you're facing one of the most perplexing infrastructure issues. Let's analyze this specific case where:

# ping 192.168.1.112
PING 192.168.1.112 (192.168.1.112) 56(84) bytes of data.
^C
--- 192.168.1.112 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2038ms

The current setup shows several potential red flags:

  • Static IP assignment (192.168.1.112/24) with proper subnet mask
  • Admin user configured with correct privileges
  • ARP responses enabled (visible in arp table)
  • No VLAN configuration

First, let's confirm basic network connectivity using tcpdump:

# tcpdump -i bond0 host 192.168.1.112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:32:45.123456 ARP, Request who-has 192.168.1.112 tell 192.168.1.100, length 28
14:32:45.123789 ARP, Reply 192.168.1.112 is-at 00:a0:a5:67:45:25, length 28

This confirms ARP resolution works but no ICMP responses.

Standard IPMI uses port 623. Let's verify:

# nmap -sU -p 623 192.168.1.112
Starting Nmap 7.80 ( https://nmap.org ) at 2023-03-28 14:34 UTC
Nmap scan report for 192.168.1.112
Host is up (0.0023s latency).

PORT    STATE  SERVICE
623/udp closed asf-rmcp

Different vendors implement IPMI differently. For Supermicro systems:

# ipmitool raw 0x30 0x70 0x0c 0
# ipmitool mc reset cold

For Dell iDRAC:

# racadm set iDRAC.NIC.Selected NIC Dedicated
# racadm reset idrac

When basic checks fail, try these advanced methods:

1. Packet Capture Analysis

# tcpdump -i bond0 -vvv -s 0 'port 623 or icmp' -w ipmi_debug.pcap

2. Firewall Rule Check

# iptables -L -n -v | grep -i 192.168.1.112

3. BMC Log Inspection

# ipmitool sel list
# ipmitool mc info

For stubborn cases requiring physical access:

  1. Power cycle the BMC (not the main system)
  2. Reset to factory defaults: 
    # ipmitool bmc reset cold
  3. Reconfigure from scratch

Remember to document all changes for future reference. This issue often resurfaces during firmware upgrades or network reconfigurations.

The configuration shows a properly assigned static IP (192.168.1.112) with correct subnet masking, yet basic ICMP connectivity fails. The ARP table successfully resolves the BMC's MAC address (00:A0:A5:67:45:25), indicating layer-2 connectivity is functioning.

# Key parameters requiring verification:
1. IP Header TTL=0x40 (may cause packet drops)
2. Default Gateway MAC shows 00:00:00:00:00:00 (invalid)
3. RMCP+ Cipher Suites enabled (0-3)
4. Per-message Auth disabled (security risk but helps troubleshooting)

First, confirm physical connectivity:

# Check switch port status (replace ethX with actual interface)
ethtool ethX | grep -E "Link detected|Speed"

Then test raw IPMI packet reception:

# Run tcpdump while attempting connection
tcpdump -i bond0 -nn -v 'host 192.168.1.112 and (port 623 or icmp)'

Path A: Gateway Configuration

# Correct the gateway MAC address via IPMI
ipmitool lan set 2 defgw ipaddr 192.168.1.1
ipmitool lan set 2 defgw macaddr 00:1A:2B:3C:4D:5E

Path B: Firewall Rules

# Example iptables rules to allow IPMI traffic
iptables -A INPUT -p udp --dport 623 -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT

For deeper analysis, run these consecutively:

# Check IPMI network statistics
ipmitool lan print 2 stats

# Verify BMC network stack
ipmitool raw 0x0c 0x02 0x01 0x00 0x00 0x00

# Test alternative authentication methods
ipmitool -I lanplus -H 192.168.1.112 -U admin -P admin chassis status

Many BMC implementations have known issues with certain firmware versions. Check release notes for:

  • ARP implementation fixes
  • ICMP responder improvements
  • RMCP+ cipher suite compatibility

Always cross-validate with vendor-specific tools when available:

# Dell iDRAC example
racadm getniccfg
racadm testping 192.168.1.1
  1. Confirm physical layer connectivity (LED status, cable tests)
  2. Validate IP configuration through alternative methods (serial console)
  3. Test with security features temporarily disabled
  4. Verify across multiple client systems

Remember to re-enable security features after successful troubleshooting:

ipmitool lan set 2 auth ADMIN PASSWORD
ipmitool lan set 2 cipher_privs aaaa