Understanding and Troubleshooting High Inode Usage in Linux Filesystems: Causes, Limits, and Solutions


2 views

Inodes are fundamental data structures in Unix/Linux filesystems that store metadata about files and directories. Each file, directory, symlink, or special device file consumes exactly one inode. When we talk about "inode usage," we're referring to the percentage of allocated inodes versus the total available inodes in the filesystem.


# Check inode usage with df
df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      6553600 512345 6041255    8% /

The maximum number of inodes is set when the filesystem is created and generally cannot be changed afterwards. The limit is determined by:

  • Filesystem type (ext4, xfs, btrfs, etc.)
  • Block size and inode ratio during mkfs
  • Total storage capacity

For ext4, you can check the maximum inodes with:


tune2fs -l /dev/sdX | grep -i inode

When inode usage reaches 100%, the system cannot create new files or directories, even if there's plenty of disk space available. You'll encounter errors like:


touch newfile.txt
touch: cannot touch 'newfile.txt': No space left on device

Despite what the error suggests, this isn't about disk space - it's specifically about inode exhaustion.

Some typical scenarios that lead to inode exhaustion:

  • Small files explosion (logs, cache, email, Docker containers)
  • Unrotated log files generating thousands of small files
  • PHP sessions or other temporary files not being cleaned up
  • Malicious activity creating thousands of files

Use these commands to identify directories with high inode consumption:


# Count files per directory
find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

# Alternative method
for i in /*; do echo $i; find $i | wc -l; done

Here are actionable approaches to resolve high inode usage:


# Clean up small files older than 30 days
find /path/to/search -type f -mtime +30 -size -10k -delete

# For mail servers with lots of small files
find /var/spool/postfix -type f -delete

# Docker cleanup (careful with this!)
docker system prune -a --volumes

For long-term prevention, consider:

  • Increasing inode count when creating new filesystems (using mkfs options)
  • Implementing proper log rotation policies
  • Monitoring inode usage proactively

Here's a simple bash script to monitor inode usage:


#!/bin/bash
THRESHOLD=90
INODE_USAGE=$(df -i | awk '{print $5}' | grep -v "Use%" | cut -d'%' -f1)

for i in $INODE_USAGE; do
    if [ $i -ge $THRESHOLD ]; then
        echo "Warning: Inode usage exceeds $THRESHOLD% on $(hostname)"
        # Add alerting logic here (email, Slack, etc.)
        exit 1
    fi
done

In Unix-like systems, an inode is a data structure that stores metadata about files and directories. Each file system has a fixed number of inodes allocated during creation, determining the maximum number of files and directories it can hold.

# Check inode usage on Linux
df -i
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1      52428800 312456 52116344    1% /

The inode limit is determined by:

  • File system type (ext4, xfs, etc.)
  • Disk partition size
  • Parameters used during file system creation (like -i bytes-per-inode in mkfs)

When inode usage reaches 100%, the system cannot create new files or directories, even if disk space is available. Common symptoms include:

# Example error messages
touch newfile.txt
touch: cannot touch 'newfile.txt': No space left on device

mkdir testdir
mkdir: cannot create directory 'testdir': No space left on device

To find directories consuming excessive inodes:

# Count files per directory (top 10)
find /path/to/search -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n | tail -10

# Alternative method using ncdu
ncdu --inodes /path/to/directory

1. Small File Accumulation

Systems generating many small files (logs, cache, sessions) often hit inode limits:

# Example: Clean PHP session files
find /var/lib/php/sessions -type f -mtime +7 -delete

# For Docker systems
docker system prune -f

2. Unrotated Log Files

Implement proper log rotation:

# Sample logrotate configuration
/var/log/app/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 644 root root
}

3. Orphaned Files

Files deleted while processes kept them open still consume inodes until the process ends:

# Find deleted but open files
lsof | grep deleted | awk '{print $2}' | sort | uniq | xargs -r kill -9
  • Monitor inode usage with tools like Nagios or Zabbix
  • Consider larger inode counts when creating file systems (mkfs.ext4 -i 8192)
  • Implement automated cleanup scripts for temporary files