How to Monitor All Newly Spawned Processes in Linux in Real-Time


3 views

When debugging process spawning behavior in Linux, traditional methods like ps aux or watch commands often miss short-lived processes. The fundamental issue lies in the polling interval - even at 0.1 second intervals, we can miss processes that complete execution faster than our observation window.

The most reliable approach taps into Linux kernel process events through these methods:

# Method 1: Using auditd (persistent logging)
sudo apt install auditd
sudo auditctl -a exit,always -F arch=b64 -S execve
sudo ausearch -sc execve -i

# Method 2: Ftrace (temporary monitoring)
echo 1 | sudo tee /sys/kernel/debug/tracing/events/sched/sched_process_exec/enable
sudo cat /sys/kernel/debug/tracing/trace_pipe

For real-time monitoring without setup overhead, Brendan Gregg's execsnoop from bcc-tools is ideal:

sudo apt install bpfcc-tools
sudo execsnoop-bpfcc -t

Sample output shows PID, PPID, and full command:

TIME(s)    PID    PPID   RET ARGS
12.345     6789   1234    0  /usr/bin/php /var/www/cron.php

For older systems without BPF support:

sudo apt install systemtap
stap -e 'probe syscall.execve { printf("%d %s\n", pid(), argstr) }'

Combine process discovery with immediate information capture:

#!/bin/bash
sudo execsnoop-bpfcc | while read -r line; do
    pid=$(echo "$line" | awk '{print $2}')
    [ -d "/proc/$pid" ] && cat "/proc/$pid/cmdline" | tr '\0' ' '
done

For production systems, note that:

  • execsnoop adds ~1ms overhead per process
  • auditd maintains persistent logs but impacts performance more significantly
  • SystemTap scripts should be pre-compiled for better efficiency

When debugging system behavior or investigating suspicious activity, we often need to monitor process creation in real-time. Traditional approaches like ps or top have fundamental limitations when dealing with short-lived processes that spawn and exit faster than our polling interval.

The most reliable approach is using Linux's audit subsystem:

# Install auditd if needed
sudo apt install auditd

# Set up process execution monitoring
sudo auditctl -a exit,always -F arch=b64 -S execve

# View the audit log (continuously)
sudo ausearch -f /bin -i | grep "proctitle"

This captures every execve() system call with full command line arguments, regardless of how quickly the process exits.

For more immediate output with less overhead, consider Brendan Gregg's execsnoop from BCC tools:

# Install BCC tools
sudo apt install bpfcc-tools

# Run execsnoop
sudo execsnoop-bpfcc

# Sample output format:
# TIME     PID     PPID    RET ARGS
# 14:23:01 12345   678     0   /usr/bin/curl example.com

For a lightweight solution that doesn't require special permissions:

#!/bin/bash
while true; do
  find /proc -maxdepth 1 -type d -name '[0-9]*' | \
    while read -r proc; do
      if [[ ! -f "${proc}/watched" ]]; then
        touch "${proc}/watched"
        cmdline=$(cat "${proc}/cmdline" | tr '\0' ' ')
        echo "$(date '+%T.%3N') ${proc##*/} ${cmdline}"
      fi
    done
  sleep 0.1
done

Each method has tradeoffs:

  • auditd: Most comprehensive but higher overhead
  • execsnoop: Near real-time with eBPF efficiency
  • inotify: Lightweight but may miss very fast processes

When monitoring production systems, you'll want to filter noise:

# Only show processes running as specific user
sudo execsnoop-bpfcc -u www-data

# Filter by parent PID
sudo execsnoop-bpfcc -p 1234