When your system reports "No such file or directory" for core utilities like /bin/ls while stat confirms their existence, you're facing one of these scenarios:

# Typical symptoms:
$ ls
-bash: /bin/ls: No such file or directory
$ stat /bin/ls
  File: /bin/ls'
  Size: 39696       Blocks: 80         IO Block: 4096   regular file
  Access: (0755/-rwxr-xr-x)  Uid: (  122/ UNKNOWN)   Gid: (  114/Debian-exim)

The immediate red flags in our case:

• UID 122 showing as "UNKNOWN" (should map to a user in /etc/passwd)
• GID 114 assigned to "Debian-exim" (completely inappropriate for system binaries)
• Recent security update preceding the issue (common attack vector)

To properly diagnose, run these commands:

# Check file integrity
$ md5sum /bin/ls
compare against known good hash from package database

# Verify package status
$ dpkg -V coreutils
??? /bin/ls

# Check loaded kernel modules
$ lsmod | grep -i 'hid'

# Examine process tree
$ ps auxf | less
look for unusual parent processes

Finding these indicators confirms a compromise:

$ rkhunter --check
[ Rootkit Hunter ] Checking for rootkits...
Checking 'basename'... Warning: Possible rootkit: 'basename' hidden process

$ strings /bin/ls | grep -A5 -B5 'lib.*so'
unusual library paths appear

Critical steps to recover:

1. Isolate the machine from network
2. Reinstall affected packages:

$ sudo apt-get --reinstall install coreutils net-tools

For comprehensive cleanup:

# Generate file change report
$ debsums -ac | grep -v 'OK$'

# Reinstall all modified packages
$ sudo apt-get install --reinstall $(dpkg -S $(debsums -ac | \
  grep -v 'OK$' | awk '{print $2}') | cut -d: -f1 | sort -u)

Essential security hardening:

# Install security tools
$ sudo apt-get install aide rkhunter chkrootkit

# Configure daily scans
$ sudo nano /etc/default/aide
AIDEARGS="--config=/etc/aide/aide.conf --check"

# Set immutable attributes on critical binaries
$ sudo chattr +i /bin/ls /bin/netstat /usr/bin/top

Remember: After a confirmed rootkit infection, the only truly secure option is complete system reinstalation from known-good media.

---

Recently I encountered a bizarre situation on an Ubuntu 8.04 server where core utilities like ls and netstat suddenly became inaccessible despite their binaries physically existing in /bin. The system would throw "-bash: /bin/ls: No such file or directory" errors while stat clearly showed the files were present.

# stat /bin/ls
  File: /bin/ls'
  Size: 39696       Blocks: 80         IO Block: 4096   regular file
Device: 803h/2051d  Inode: 1073910881  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  122/ UNKNOWN)   Gid: (  114/Debian-exim)

The first suspicious sign was the unusual UID/GID assignments. Neither 122 nor 114 existed in /etc/passwd, which immediately suggested possible system compromise. The Debian-exim group name appearing for a core utility was particularly concerning.

To verify the integrity of binaries, I used:

# ls -laZ /bin/ls
# dpkg -V coreutils

1. Check loaded libraries:

# ldd /bin/ls
# cat /etc/ld.so.preload

2. Verify package integrity:

# debsums -a
# apt-get install --reinstall coreutils net-tools

3. Rootkit detection:

# rkhunter --check
# chkrootkit

After confirming a rootkit through rkhunter, here's the immediate containment process:

1. Disconnect the machine from network
2. Boot from live CD/USB
3. Mount the filesystem read-only
4. Collect forensic evidence
5. Reinstall core packages:

# mount -o remount,ro /
# apt-get update && apt-get --reinstall install \
    coreutils procps net-tools bash
# updatedb && ldconfig

This incident highlighted several critical security practices:

• Regularly verify package integrity with debsums
• Monitor for unusual UID/GID assignments
• Implement tripwire or aide for file integrity checking
• Maintain offline backups of critical systems

For production systems, consider implementing:

# cron job for daily integrity checks
0 3 * * * /usr/bin/debsums -csa | mail -s "Daily debsums report" admin@example.com