Debugging NRPE “Unable to read output” Error: Root Causes and Solutions


1 views

When dealing with NRPE (Nagios Remote Plugin Executor), the "Unable to read output" error can be particularly frustrating because it masks the actual underlying issue. Let's break down the problem based on the symptoms:

# On NRPE client (working):
/usr/lib/nagios/plugins/additional/check_openmanage -s -e -b ctrl_driver=0 bat_charge

# From Nagios server (failing):
/usr/local/nagios/plugins/check_nrpe -H comxps -c check_openmanage
NRPE: Unable to read output

The log message about being unable to open nrpe.cfg is actually a red herring. NRPE does read the file eventually, but the warning indicates either:

  • Permission issues on the config file
  • Multiple config file locations being checked
  • SELinux context problems on RHEL

Verify the actual config file being used with:

ps aux | grep nrpe
# Check the --config option in the running process

The switch from RHEL 4 to RHEL 5 introduced several changes that could affect NRPE:

  1. SELinux is more restrictive by default
  2. Different library paths
  3. Changed default permissions model

For SELinux, try these diagnostics:

# Check SELinux status
getenforce

# Check for denials
grep nrpe /var/log/audit/audit.log
# or on newer systems:
journalctl -t setroubleshoot

The most common root cause for "Unable to read output" is that the plugin executed but:

  • Exited too quickly before NRPE could capture output
  • Produced output but exited with error
  • Had environment differences between manual and NRPE execution

Create a test wrapper script to log execution details:

#!/bin/bash
# /usr/lib/nagios/plugins/additional/debug_wrapper.sh

logfile="/tmp/nrpe_debug.log"
echo "==== $(date) ====" >> $logfile
echo "Environment:" >> $logfile
env >> $logfile
echo "Command: $@" >> $logfile
/usr/lib/nagios/plugins/additional/check_openmanage "$@" 2>&1 | tee -a $logfile
exit ${PIPESTATUS[0]}

Update your nrpe.cfg to use the wrapper:

command[check_openmanage]=/usr/lib/nagios/plugins/additional/debug_wrapper.sh -s -e -b ctrl_driver=0 bat_charge

Network issues between Nagios server and NRPE client can manifest as output reading problems. Test with:

# Increase timeout for testing
check_nrpe -H comxps -t 60 -c check_openmanage

# Check network path
traceroute comxps
mtr comxps

When all else fails, use these advanced methods:

1. Strace the NRPE process:

strace -f -p $(pgrep nrpe) -s 999 -o /tmp/nrpe_strace.log

2. Packet capture:

tcpdump -i any port 5666 -w /tmp/nrpe.pcap

3. Test with minimal configuration:

# Minimal nrpe.cfg for testing
allowed_hosts=your.nagios.server.ip
dont_blame_nrpe=0
command[test]=/bin/echo Hello World

Before giving up, verify these often-overlooked items:

  • User/group permissions on all plugin scripts
  • LD_LIBRARY_PATH differences between users
  • Missing dependencies in the plugin
  • Firewall rules on both ends
  • Nagios server's NRPE version compatibility with client

Remember that NRPE errors can be deceptive - the "Unable to read output" message often means the plugin failed in a way that NRPE couldn't properly report. Methodical elimination of variables is key to resolving these issues.

When dealing with NRPE (Nagios Remote Plugin Executor) configurations, the "Unable to read output" error is particularly frustrating because it gives little indication of the actual problem. Here's what we're seeing:

/usr/local/nagios/plugins/check_nrpe -H nrpeclient → NRPE v2.12 (expected)
/usr/local/nagios/plugins/check_nrpe -H comxps -c check_openmanage → NRPE: Unable to read output

Before deep diving, let's verify basic functionality:

# On the NRPE client:
ps aux | grep nrpe
netstat -tulnp | grep 5666
/usr/lib/nagios/plugins/additional/check_openmanage -s -e -b ctrl_driver=0 bat_charge

1. File Permissions

RHEL 5's SELinux can cause silent failures. Check:

ls -lZ /usr/lib/nagios/plugins/additional/check_openmanage
getenforce
# Temporary test:
setenforce 0
# Then retest NRPE

2. NRPE Configuration

The error about unable to open config file is suspicious despite it reading the file. Check for:

# In nrpe.cfg:
dont_blame_nrpe=1
allowed_hosts=127.0.0.1,your.nagios.server
command_timeout=60

3. Command Execution Context

Test if NRPE can actually execute the plugin:

sudo -u nagios /usr/lib/nagios/plugins/additional/check_openmanage -s -e -b ctrl_driver=0 bat_charge

When basic checks don't reveal the issue:

# On the NRPE server:
strace -f -o /tmp/nrpe.strace /usr/local/nagios/bin/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d

# On the Nagios server:
tcpdump -i eth0 -nn -s0 -w /tmp/nrpe_debug.pcap port 5666

For RHEL 5 systems, pay special attention to:

  • Older OpenSSL versions causing handshake failures
  • SELinux policies needing updates for NRPE
  • Possible library path issues with 32-bit vs 64-bit plugins

Here's a verified working nrpe.cfg snippet:

# NRPE Configuration
log_facility=daemon
pid_file=/var/run/nrpe/nrpe.pid
server_port=5666
nrpe_user=nagios
nrpe_group=nagios
allowed_hosts=192.168.1.100
dont_blame_nrpe=1
command_timeout=60
debug=1

# Command definitions
command[check_openmanage]=/usr/lib/nagios/plugins/additional/check_openmanage -s -e -b ctrl_driver=0 bat_charge

After making changes:

service nrpe restart
tail -f /var/log/messages
/usr/local/nagios/libexec/check_nrpe -H localhost -c check_openmanage