Troubleshooting God Process Monitoring on CentOS: Fixing “Event System Not Found” Error


5 views

When setting up God process monitoring on two new CentOS 5.5 x86_64 servers, I encountered a perplexing issue where the event system failed to load:

$ tail /var/log/god.log 
E [2011-04-22 12:33:17] ERROR: Condition 'God::Conditions::ProcessExits' 
requires an event system but none has been loaded 

$ god check 
using event system: none 
[fail] event system did not load 

What's particularly strange is that I have identical CentOS 5.5 installations where God works perfectly:

$ god check 
using event system: netlink 
starting event handler 
forking off new process 
forked process with pid = 17559 
killing process 
[ok] process exit event received 

The key difference appears in the error message when God attempts to load the Netlink event handler:

no such file to load -- netlink_handler_ext 

This suggests the Ruby extension for Netlink support is missing. After investigating, I found that while all servers run:

ruby 1.8.6 (2010-02-05 patchlevel 399) [x86_64-linux]

The working systems had the netlink_handler_ext native extension properly installed, while the new systems didn't.

The fix involves properly installing God with its native extensions:

# Uninstall existing God gem
gem uninstall god

# Install development tools
yum groupinstall "Development Tools"
yum install ruby-devel

# Reinstall God with native extensions
gem install god --no-ri --no-rdoc

After reinstalling, verify the event system works:

$ god check
using event system: netlink 
starting event handler 
forking off new process 
forked process with pid = 28741 
killing process 
[ok] process exit event received

If the above doesn't work, consider these alternatives:

# Option 1: Force rebuild of native extensions
gem pristine god

# Option 2: Install from source
git clone https://github.com/mojombo/god.git
cd god
rake install

The problem ultimately stems from how Ruby native extensions are built during installation. The netlink_handler_ext is a C extension that provides low-level process monitoring capabilities. When development tools or headers are missing during gem installation, this critical component fails to compile.


When trying to run God process monitoring on newly provisioned CentOS 5.5 x86_64 servers, you encounter the following error:

$ tail /var/log/god.log 
E [2011-04-22 12:33:17] ERROR: Condition 'God::Conditions::ProcessExits' 
requires an event system but none has been loaded 

$ god check 
using event system: none 
[fail] event system did not load

Interestingly, other CentOS 5.5 servers with identical configurations work perfectly:

$ god check 
using event system: netlink 
starting event handler 
forking off new process 
forked process with pid = 17559 
killing process 
[ok] process exit event received

After examining both environments, several factors emerge:

  • Identical Ruby version (1.8.6-p399 from ELFF repo)
  • Same CentOS version (5.5 x86_64)
  • Different kernel versions:
    Working: 2.6.18-194.el5
    Problematic: 2.6.18-238.9.1.el5

The error occurs when God attempts to load the netlink event handler extension:

no such file to load -- netlink_handler_ext 

This typically happens when:

  1. The netlink kernel module isn't available
  2. The Ruby netlink extension wasn't compiled properly
  3. Kernel headers are missing during gem installation

1. Verify Kernel Modules

First check if netlink is available in your kernel:

lsmod | grep netlink
modprobe netlink

2. Reinstall God with Proper Dependencies

Uninstall and reinstall with development tools:

yum groupinstall "Development Tools"
yum install kernel-devel

gem uninstall god
gem install god -- --with-cflags=\"-O2 -pipe -march=nocona\"

3. Alternative Event Systems

If netlink still fails, try forcing another event system:

# In your god config:
God.contact(:email) do |c|
  # ... email settings ...
end

# Force poll instead of event system
God::EventHandler::EVENTS[:poll] = true

4. Kernel Version Fallback

If the issue persists, consider:

# Downgrade kernel to match working servers
yum install kernel-2.6.18-194.el5

# Or compile custom netlink module
wget https://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.gz
tar xzvf linux-2.6.18.tar.gz
cd linux-2.6.18/net/netlink
make -C /lib/modules/$(uname -r)/build M=$(pwd) modules

After applying fixes, verify with:

god check
# Should now show:
using event system: netlink
starting event handler