Server Breach Response Guide: Forensic Analysis, Recovery & Mitigation for Linux Admins


4 views

When facing a compromised server, execute this emergency checklist:

# Isolate network immediately
sudo iptables -P INPUT DROP
sudo iptables -P OUTPUT DROP
sudo systemctl stop networking

# Capture volatile forensic data
sudo netstat -tulnpe > /tmp/netstat_snapshot.txt
sudo lsof -i > /tmp/lsof_snapshot.txt
sudo ps auxef > /tmp/process_snapshot.txt

For thorough root cause analysis:

# Find recently modified files (last 48 hours)
find / -type f -mtime -2 -print0 | xargs -0 ls -lt | tee /tmp/recent_files.txt

# Check for suspicious crontabs
for user in $(cut -f1 -d: /etc/passwd); do 
    crontab -u $user -l 2>/dev/null | grep -v "^#"
done > /tmp/all_crontabs.txt

# Detect hidden processes
ps -eo pid,user,cmd | grep -E "(\./|/tmp|/dev/shm)"

After containment:

  1. Build clean server from known-good backups
  2. Implement strict access controls:
# Harden SSH configuration
echo "PermitRootLogin no" >> /etc/ssh/sshd_config
echo "PasswordAuthentication no" >> /etc/ssh/sshd_config
echo "AllowUsers admin_user" >> /etc/ssh/sshd_config
systemctl restart sshd

Essential security measures:

# Install and configure auditd
sudo apt install auditd
sudo auditctl -a exit,always -F arch=b64 -S execve -k process_execution
sudo auditctl -w /var/www/html -p wa -k web_content

Example OSSEC rule for detecting web shells:

<rule id="100101" level="10">
    <if_sid>1000</if_sid>
    <match>eval\(|base64_decode|shell_exec|passthru</match>
    <description>Potential web shell detected</description>
</rule>

Essential fields to log:

{
    "timestamp": "ISO-8601 format",
    "initial_vector": "phishing/webapp/RCE/etc",
    "compromise_scope": ["files_changed", "data_accessed"],
    "containment_time": "duration",
    "recovery_steps": ["step1", "step2"],
    "preventative_measures": ["2FA_enabled", "WAF_rules"]
}

When you suspect a server breach, time is critical. Here's what to do:

  1. Isolate the system: Disconnect from the network immediately to prevent lateral movement or ongoing attacks
    # On Linux:
    sudo ifconfig eth0 down
    # Or more drastically:
    sudo shutdown -h now
  2. Preserve evidence before making changes:
    # Create forensic copies of critical logs
    sudo cp -a /var/log /secure/backup/logs_backup
    # Capture process information
    ps auxf > /secure/backup/process_snapshot.txt
    netstat -tulnp > /secure/backup/network_connections.txt

In the case study mentioned, the breach occurred through:

  • Unsecured file upload in ZenCart (CVE-2010-2161)
  • Lack of file extension validation
  • Missing authentication checks

To detect similar vulnerabilities in PHP applications:

// Secure file upload example
function validateUpload($file) {
    $allowed = ['jpg', 'png', 'gif'];
    $ext = pathinfo($file['name'], PATHINFO_EXTENSION);
    
    if (!in_array(strtolower($ext), $allowed)) {
        throw new Exception('Invalid file type');
    }
    
    if (!getimagesize($file['tmp_name'])) {
        throw new Exception('File is not a valid image');
    }
    
    // Additional checks:
    // - File size limits
    // - Virus scanning
    // - Authentication/authorization
}

Follow this methodology for secure recovery:

# 1. Create a clean snapshot of current state
sudo dd if=/dev/sda of=/mnt/secure/forensic_image.img bs=4M

# 2. Rebuild from known good sources
# For package managers:
sudo apt-get clean
sudo apt-get update
sudo apt-get install --reinstall $(dpkg -l | grep ^ii | awk '{print $2}')

# 3. Implement monitoring
sudo apt-get install auditd
sudo auditctl -a exit,always -F arch=b64 -S execve -k process_execution

Key security measures to implement:

  • File integrity monitoring:
    # Using AIDE (Advanced Intrusion Detection Environment)
    sudo aideinit
    sudo aide --check
  • Automated vulnerability scanning:
    # Using Lynis for Linux systems
    sudo lynis audit system

Essential components should include:

# Sample IR checklist template
{
  "immediate_actions": [
    "Isolate affected systems",
    "Preserve evidence",
    "Notify stakeholders"
  ],
  "investigation": [
    "Identify IOC (Indicators of Compromise)",
    "Determine attack vector",
    "Assess data exposure"
  ],
  "recovery": [
    "Patch vulnerabilities",
    "Reset credentials",
    "Restore from clean backups"
  ],
  "post_mortem": [
    "Document timeline",
    "Identify process improvements",
    "Update security policies"
  ]
}

Remember to regularly test your incident response procedures through tabletop exercises and red team engagements.