The key to customizing Nagios notifications lies in the service_notification_commands
parameter and proper contact/contactgroup configuration. The original setup simultaneously triggers both email and pager for all alerts, which isn't ideal for tiered response scenarios.
We need to implement a two-layer notification system:
- Email notifications for WARNING state changes
- Pager notifications exclusively for CRITICAL state changes
First, create separate notification commands in your Nagios configuration:
# Email-only notification command
define command {
command_name notify-service-by-email-only
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}
# Pager-only notification command
define command {
command_name notify-service-by-pager-only
command_line /usr/bin/printf "%b" "Service: $SERVICEDESC$\nHost: $HOSTALIAS$\nState: $SERVICESTATE$\nTime: $SHORTDATETIME$\nInfo: $SERVICEOUTPUT$" | /usr/bin/pager_command $CONTACTPAGER$
}
Modify your contact definition to implement tiered notifications:
define contact {
name tiered-contact
service_notification_options w,u,c,r,f,s
host_notification_options d,u,r,f,s
service_notification_commands notify-service-by-email-only,notify-service-by-pager-only
host_notification_commands notify-host-by-email,notify-host-by-pager
register 0
service_notification_period 24x7
host_notification_period 24x7
}
To achieve the conditional behavior, we'll use Nagios' notification escalation features:
define serviceescalation {
host_name *
service_description *
first_notification 1
last_notification 0
notification_interval 0
contact_groups pager-admins
escalation_options c
escalation_period 24x7
}
After implementing these changes:
- Restart Nagios to apply configuration changes
- Test with
service nagios configtest
first - Force a WARNING state alert to verify email-only delivery
- Force a CRITICAL state alert to verify both email and pager delivery
For more granular control, consider these additional parameters:
define contact {
# ... existing parameters ...
service_notification_options w
host_notification_options d,u
}
define contactgroup {
contactgroup_name email-admins
alias Email Administrators
members admin1,admin2
service_notification_commands notify-service-by-email-only
host_notification_commands notify-host-by-email
}
In enterprise monitoring systems, differentiating notification channels based on alert severity is crucial. The requirement is to modify Nagios' default behavior where both email and pager notifications fire for all service states (WARNING, CRITICAL, etc.). Instead, we want:
- Email notifications for WARNING and CRITICAL states
- Pager notifications ONLY for CRITICAL states
This requires modifying both contact definitions and notification commands. Here's the technical approach:
define command {
command_name notify-service-critical-pager
command_line /usr/bin/printf "%b" "$NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTPAGER$
}
define command {
command_name notify-service-warning-email
command_line /usr/bin/printf "%b" "$NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Alert - $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}
The contact definition needs to reference these custom commands:
define contact {
name priority-contact
service_notification_options w,u,c,r,f,s
host_notification_options d,u,r,f,s
service_notification_commands notify-service-warning-email,notify-service-critical-pager
host_notification_commands notify-host-by-email
register 0
service_notification_period 24x7
host_notification_period 24x7
}
For more complex scenarios, consider using Nagios event handlers or modifying notification scripts:
#!/bin/bash
# Custom notification script: notify_custom.sh
case "$SERVICESTATE" in
"CRITICAL")
# Send both email and pager
echo "$NOTIFICATIONTYPE$ Alert for $HOSTALIAS$/$SERVICEDESC$" | mail -s "CRITICAL Alert" $CONTACTEMAIL$
echo "$NOTIFICATIONTYPE$ Alert for $HOSTALIAS$/$SERVICEDESC$" | mail -s "CRITICAL Alert" $CONTACTPAGER$
;;
"WARNING")
# Send only email
echo "$NOTIFICATIONTYPE$ Alert for $HOSTALIAS$/$SERVICEDESC$" | mail -s "WARNING Alert" $CONTACTEMAIL$
;;
esac
After implementation:
- Force a WARNING state (e.g., set disk threshold temporarily)
- Verify only email notification is received
- Force a CRITICAL state
- Verify both email and pager notifications are received