When troubleshooting system issues or investigating past events, the standard ps
command falls short because it only shows currently running processes. The Linux kernel doesn't maintain a permanent process database by default, but several tools and techniques can help reconstruct process history.
Many Linux distributions include the process accounting package (often called acct
or psacct
) which logs process activity:
# Install process accounting (Debian/Ubuntu)
sudo apt install acct
# Start the service
sudo systemctl enable --now psacct
# View terminated processes
sudo lastcomm -f /var/log/account/pacct
Sample output showing terminated processes:
sshd john pts/0 0.00 secs Tue Oct 10 15:30
bash jane pts/1 0.05 secs Tue Oct 10 15:28 (killed)
python3 root __ 10.23 secs Tue Oct 10 15:25
The Linux audit system provides granular process tracking:
# Install auditd
sudo apt install auditd
# Configure process exec tracking
sudo auditctl -a exit,always -F arch=b64 -S execve
# View process execution logs
sudo ausearch -sc execve -i
For systems using systemd, the journal may contain process information:
# View process start/stop events
journalctl -u systemd-journald --since "1 hour ago" | grep -E "Started|Stopped"
# Filter for specific process
journalctl _COMM=bash --since "09:00" --until "10:00"
If you anticipate needing process history, consider periodic snapshots:
# Create a cron job to log current processes
*/5 * * * * ps -eo pid,ppid,cmd,start_time > /var/log/process_snapshot_$(date +\%Y\%m\%d-\%H\%M).log
The perf
tool can capture process lifecycle events:
# Capture process create/exit events
sudo perf record -e sched:sched_process_exec -e sched:sched_process_exit -a
# Generate report
sudo perf script
For comprehensive investigations, consider specialized tools:
# Install and run lynis for system audit
sudo apt install lynis
sudo lynis audit system
Each method has different retention periods and overhead. For immediate troubleshooting, journalctl
and auditd
typically offer the best balance between detail and accessibility.
When investigating system events or performance issues, simply viewing currently running processes (via ps
) often isn't sufficient. Unlike Windows systems that maintain process histories, Linux typically requires proactive monitoring solutions or log analysis.
Many Linux distributions include process accounting tools that log process activity:
# Install process accounting (Debian/Ubuntu)
sudo apt install acct
# Enable accounting
sudo systemctl enable --now psacct
# View terminated processes (last 50)
lastcomm | head -50
# Filter by specific command
lastcomm -f /usr/bin/python3
The lastcomm
output shows:
- Process name
- User who ran it
- Termination status
- CPU time used
- Execution timestamp
For systems using systemd (most modern distributions):
# View all process starts/stops in last hour
journalctl --since "1 hour ago" -u systemd-journald | grep -E 'Started|Stopped'
# Filter by specific service
journalctl -u nginx --since "09:00" --until "10:00"
For more detailed auditing:
# Install audit framework
sudo apt install auditd
# Track process executions
sudo auditctl -a exit,always -F arch=b64 -S execve
# View audit logs (process executions with timestamps)
ausearch -sc execve -ts today
For ad-hoc monitoring when you anticipate needing process history:
#!/bin/bash
# Continuous process snapshot logger
while true; do
timestamp=$(date +"%Y-%m-%d_%H:%M:%S")
ps -eo pid,ppid,cmd,%mem,%cpu,etime,lstart > /var/log/process_snapshot_${timestamp}.log
sleep 300 # 5 minute intervals
done
For comprehensive historical tracking consider:
- atop: Advanced system monitor with logging (
atop -r /var/log/atop/atop_20240301
) - sysdig: System exploration tool (
sysdig -w trace.scap
to capture) - prometheus+node_exporter: For long-term metrics
To find all Apache processes that terminated abnormally in the last 2 hours:
journalctl --since "2 hours ago" | grep apache2 | grep -i -E 'failed|killed|terminated'
# Cross-reference with accounting data
lastcomm -f /usr/sbin/apache2 | grep -v "still running"