How to View Terminated/Killed Process History in Linux: Forensic Process Investigation Techniques


5 views

When troubleshooting system issues or investigating past events, the standard ps command falls short because it only shows currently running processes. The Linux kernel doesn't maintain a permanent process database by default, but several tools and techniques can help reconstruct process history.

Many Linux distributions include the process accounting package (often called acct or psacct) which logs process activity:

# Install process accounting (Debian/Ubuntu)
sudo apt install acct

# Start the service
sudo systemctl enable --now psacct

# View terminated processes
sudo lastcomm -f /var/log/account/pacct

Sample output showing terminated processes:

sshd          john     pts/0      0.00 secs Tue Oct 10 15:30
bash          jane     pts/1      0.05 secs Tue Oct 10 15:28 (killed)
python3       root     __         10.23 secs Tue Oct 10 15:25

The Linux audit system provides granular process tracking:

# Install auditd
sudo apt install auditd

# Configure process exec tracking
sudo auditctl -a exit,always -F arch=b64 -S execve

# View process execution logs
sudo ausearch -sc execve -i

For systems using systemd, the journal may contain process information:

# View process start/stop events
journalctl -u systemd-journald --since "1 hour ago" | grep -E "Started|Stopped"

# Filter for specific process
journalctl _COMM=bash --since "09:00" --until "10:00"

If you anticipate needing process history, consider periodic snapshots:

# Create a cron job to log current processes
*/5 * * * * ps -eo pid,ppid,cmd,start_time > /var/log/process_snapshot_$(date +\%Y\%m\%d-\%H\%M).log

The perf tool can capture process lifecycle events:

# Capture process create/exit events
sudo perf record -e sched:sched_process_exec -e sched:sched_process_exit -a

# Generate report
sudo perf script

For comprehensive investigations, consider specialized tools:

# Install and run lynis for system audit
sudo apt install lynis
sudo lynis audit system

Each method has different retention periods and overhead. For immediate troubleshooting, journalctl and auditd typically offer the best balance between detail and accessibility.


When investigating system events or performance issues, simply viewing currently running processes (via ps) often isn't sufficient. Unlike Windows systems that maintain process histories, Linux typically requires proactive monitoring solutions or log analysis.

Many Linux distributions include process accounting tools that log process activity:

# Install process accounting (Debian/Ubuntu)
sudo apt install acct

# Enable accounting
sudo systemctl enable --now psacct

# View terminated processes (last 50)
lastcomm | head -50

# Filter by specific command
lastcomm -f /usr/bin/python3

The lastcomm output shows:

  • Process name
  • User who ran it
  • Termination status
  • CPU time used
  • Execution timestamp

For systems using systemd (most modern distributions):

# View all process starts/stops in last hour
journalctl --since "1 hour ago" -u systemd-journald | grep -E 'Started|Stopped'

# Filter by specific service
journalctl -u nginx --since "09:00" --until "10:00"

For more detailed auditing:

# Install audit framework
sudo apt install auditd

# Track process executions
sudo auditctl -a exit,always -F arch=b64 -S execve

# View audit logs (process executions with timestamps)
ausearch -sc execve -ts today

For ad-hoc monitoring when you anticipate needing process history:

#!/bin/bash
# Continuous process snapshot logger
while true; do
    timestamp=$(date +"%Y-%m-%d_%H:%M:%S")
    ps -eo pid,ppid,cmd,%mem,%cpu,etime,lstart > /var/log/process_snapshot_${timestamp}.log
    sleep 300  # 5 minute intervals
done

For comprehensive historical tracking consider:

  • atop: Advanced system monitor with logging (atop -r /var/log/atop/atop_20240301)
  • sysdig: System exploration tool (sysdig -w trace.scap to capture)
  • prometheus+node_exporter: For long-term metrics

To find all Apache processes that terminated abnormally in the last 2 hours:

journalctl --since "2 hours ago" | grep apache2 | grep -i -E 'failed|killed|terminated'

# Cross-reference with accounting data
lastcomm -f /usr/sbin/apache2 | grep -v "still running"