The first red flag you mentioned - abnormal network traffic from random high ports - is indeed concerning. Let's start by mapping these connections to processes:
# Comprehensive network process mapping sudo netstat -tulnp sudo ss -tulnp # For more detailed socket information sudo lsof -i -P -n
Standard process viewers can be bypassed by rootkits. Here's how to perform deeper analysis:
# Compare different process listing methods ps auxf top -H -b -n 1 htop # Check for discrepancies between /proc and process lists ls -la /proc/[0-9]*/exe | awk '{print $11}' | sort | uniq
Rootkits often hide their files using various techniques. Try these detection methods:
# Find unlinked (hidden) but still running processes sudo grep -l ' (deleted)' /proc/*/exe # Check for suspicious hidden directories sudo find / -type d -name ".*" -print # Verify critical binaries for bin in $(which ls ps netstat sshd); do rpm -Vf $bin || echo "$bin modified"; done
Malicious kernel modules (LKM rootkits) require special attention:
# List loaded modules lsmod # Compare with expected modules cat /proc/modules # Check module signatures (if enabled) sudo modinfo $(lsmod | awk '{print $1}' | tail -n +2)
Volatile memory analysis can reveal hidden processes:
# Install and use LiME for memory acquisition sudo apt-get install lime-forensics sudo insmod ./lime-forensics.ko "path=/tmp/memdump.lime format=lime" # Then analyze with Volatility Framework vol.py -f /tmp/memdump.lime linux_pslist
For future protection, implement these monitoring solutions:
# Auditd configuration for process monitoring sudo apt-get install auditd sudo auditctl -a exit,always -F arch=b64 -S execve sudo auditctl -a exit,always -F arch=b32 -S execve # Monitor critical directories sudo auditctl -w /bin/ -p wa -k system_binaries sudo auditctl -w /sbin/ -p wa -k system_sbinaries
Set up automated integrity verification with tools like AIDE:
# Initial setup sudo apt-get install aide sudo aideinit # Daily check via cron 0 0 * * * /usr/bin/aide --check | mail -s "AIDE Report" admin@example.com
If you confirm a compromise, follow these steps:
# Capture forensic evidence first sudo dd if=/dev/sda of=/mnt/evidence/sda.img bs=4M # Then isolate the system sudo iptables -A INPUT -j DROP sudo iptables -A OUTPUT -j DROP
When noticing abnormal network patterns like random high-port connections, your first instinct should be to suspect process hiding techniques. Common indicators include:
- Unexpected TCP/UDP connections on ephemeral ports (32768-60999)
- CPU/memory usage discrepancies between system monitors and process lists
- Modified timestamps on critical system binaries (ls, ps, netstat)
Begin with these diagnostic commands to identify inconsistencies:
# Compare process lists from different sources
ps aux
ls -la /proc/*/exe 2>/dev/null | grep deleted
# Network connection verification
netstat -tulnp
ss -tulnp
lsof -i -P -n
# Loaded kernel module check
lsmod
cat /proc/modules
For sophisticated rootkits that hook system calls, use these methods:
# Check for LD_PRELOAD hijacking
cat /proc/*/environ | grep LD_PRELOAD
# Verify system call tables
grep sys_call_table /boot/System.map-$(uname -r)
# Use statically compiled tools
busybox ps
busybox netstat
Implement these ongoing monitoring strategies:
# Cron job for process auditing
*/5 * * * * root /usr/bin/diff <(ps -eo pid,cmd) <(/bin/ps -eo pid,cmd)
# Kernel module monitoring
#!/bin/bash
while true; do
diff <(lsmod | sort) <(cat /tmp/lsmod.baseline | sort)
sleep 300
done
If compromise is confirmed:
- Isolate the machine immediately
- Capture volatile memory (using LiME or fmem)
- Create disk images for forensic analysis
- Rotate all credentials that may have been exposed