When your application unexpectedly receives SIGTERM signals in RHEL 5 environments, identifying the source becomes critical for system stability. Unlike modern Linux kernels, RHEL 5's older toolchain requires specific approaches for signal tracing.

Begin with kernel-level auditing using the system audit framework:

# Configure audit rules for SIGTERM
auditctl -a exit,always -F arch=b32 -S kill -S tkill -S tgkill -F a1=15

This captures all processes sending SIGTERM (signal 15). Review logs with ausearch -sc kill or check /var/log/audit/audit.log.

For real-time monitoring of a specific process:

strace -p [PID] -e trace=signal -s 100 -o /tmp/signal_trace.log

This logs all signals received by the process, including sender details when available.

For deeper analysis, use SystemTap to create a signal tracing script:

probe kernel.function("send_signal") {
    if (sig == 15) {  // SIGTERM
        printf("SIGTERM sent by PID %d (%s) to PID %d\n", 
            pid(), execname(), sig_pid)
    }
}

Run with stap script.stp after installing kernel debuginfo packages.

Enhance your signal handler to capture more context:

void term_handler(int sig) {
    char buf[256];
    sprintf(buf, "/proc/%d/status", getppid());
    FILE *f = fopen(buf, "r");
    // Log parent process info
    // Also consider logging /proc/self/status
    // and backtrace information
}

Enable process accounting for historical tracking:

# Enable accounting
/etc/init.d/psacct start
# Review command history
lastcomm -f /var/account/pacct | grep [your_app_name]

For persistent issues, consider LD_PRELOAD interception:

// sigtrace.c
int kill(pid_t pid, int sig) {
    if (sig == SIGTERM) {
        log_origin(getpid(), pid);
    }
    return real_kill(pid, sig);
}
// Compile with:
// gcc -shared -fPIC -o sigtrace.so sigtrace.c -ldl
// Then run:
// LD_PRELOAD=./sigtrace.so your_application

When your application suddenly receives SIGTERM signals with no apparent source, it's like debugging ghost behavior in your system. In RHEL 5 environments, several potential origins exist for these termination signals.

auditd provides powerful monitoring capabilities:

# Install audit tools if missing
yum install audit

# Configure audit for signal monitoring
auditctl -a exit,always -F arch=b32 -S kill -S tkill -S tgkill
auditctl -a exit,always -F arch=b64 -S kill -S tkill -S tgkill

# View audit logs
ausearch -sc kill -i

Combine these approaches for maximum visibility:

1. SystemTap Scripting

probe signal.send {
  if (sig_name == "SIGTERM") {
    printf("%s sent SIGTERM to %s (pid: %d)\n", execname(), pid_name, sig_pid)
  }
}

2. Kernel-Level Monitoring

# Trace all signal deliveries system-wide
strace -f -e trace=signal -p $(pgrep your_app) 2>&1 | grep SIGTERM

Enhance your signal handler to log additional context:

void sigterm_handler(int sig) {
    FILE *fp = fopen("/var/log/app_signals.log", "a");
    if (fp) {
        fprintf(fp, "SIGTERM received at %ld from PID %d (parent %d)\n",
                (long)time(NULL), getppid(), getppid());
        fclose(fp);
    }
    // Normal cleanup continues...
}

• Legacy monitoring tools (pre-systemd init scripts)
• Cron job timeouts
• Resource monitoring daemons
• Interactive shell sessions sending signals

For persistent issues, consider low-level tracing:

# Load kernel debug symbols first
perf probe --add 'kernel.do_send_specific sig=%di pid=%si'