When debugging process spawning behavior in Linux, traditional methods like ps aux
or watch
commands often miss short-lived processes. The fundamental issue lies in the polling interval - even at 0.1 second intervals, we can miss processes that complete execution faster than our observation window.
The most reliable approach taps into Linux kernel process events through these methods:
# Method 1: Using auditd (persistent logging)
sudo apt install auditd
sudo auditctl -a exit,always -F arch=b64 -S execve
sudo ausearch -sc execve -i
# Method 2: Ftrace (temporary monitoring)
echo 1 | sudo tee /sys/kernel/debug/tracing/events/sched/sched_process_exec/enable
sudo cat /sys/kernel/debug/tracing/trace_pipe
For real-time monitoring without setup overhead, Brendan Gregg's execsnoop from bcc-tools is ideal:
sudo apt install bpfcc-tools
sudo execsnoop-bpfcc -t
Sample output shows PID, PPID, and full command:
TIME(s) PID PPID RET ARGS 12.345 6789 1234 0 /usr/bin/php /var/www/cron.php
For older systems without BPF support:
sudo apt install systemtap
stap -e 'probe syscall.execve { printf("%d %s\n", pid(), argstr) }'
Combine process discovery with immediate information capture:
#!/bin/bash
sudo execsnoop-bpfcc | while read -r line; do
pid=$(echo "$line" | awk '{print $2}')
[ -d "/proc/$pid" ] && cat "/proc/$pid/cmdline" | tr '\0' ' '
done
For production systems, note that:
- execsnoop adds ~1ms overhead per process
- auditd maintains persistent logs but impacts performance more significantly
- SystemTap scripts should be pre-compiled for better efficiency
When debugging system behavior or investigating suspicious activity, we often need to monitor process creation in real-time. Traditional approaches like ps
or top
have fundamental limitations when dealing with short-lived processes that spawn and exit faster than our polling interval.
The most reliable approach is using Linux's audit subsystem:
# Install auditd if needed
sudo apt install auditd
# Set up process execution monitoring
sudo auditctl -a exit,always -F arch=b64 -S execve
# View the audit log (continuously)
sudo ausearch -f /bin -i | grep "proctitle"
This captures every execve() system call with full command line arguments, regardless of how quickly the process exits.
For more immediate output with less overhead, consider Brendan Gregg's execsnoop from BCC tools:
# Install BCC tools
sudo apt install bpfcc-tools
# Run execsnoop
sudo execsnoop-bpfcc
# Sample output format:
# TIME PID PPID RET ARGS
# 14:23:01 12345 678 0 /usr/bin/curl example.com
For a lightweight solution that doesn't require special permissions:
#!/bin/bash
while true; do
find /proc -maxdepth 1 -type d -name '[0-9]*' | \
while read -r proc; do
if [[ ! -f "${proc}/watched" ]]; then
touch "${proc}/watched"
cmdline=$(cat "${proc}/cmdline" | tr '\0' ' ')
echo "$(date '+%T.%3N') ${proc##*/} ${cmdline}"
fi
done
sleep 0.1
done
Each method has tradeoffs:
- auditd: Most comprehensive but higher overhead
- execsnoop: Near real-time with eBPF efficiency
- inotify: Lightweight but may miss very fast processes
When monitoring production systems, you'll want to filter noise:
# Only show processes running as specific user
sudo execsnoop-bpfcc -u www-data
# Filter by parent PID
sudo execsnoop-bpfcc -p 1234