You're seeing this frustrating situation where system services fail to restart with "No space left on device" errors, yet your disk monitoring tools show plenty of available space and inodes. Here's what's really happening under the hood.

$ service apache2 restart
Error: No space left on device

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       443G   28G  393G   7% /

Modern Linux systems (especially after Debian 8) use mount namespaces for service isolation. When you see available space in df but services fail, you're probably hitting one of these less obvious limits:

• Private /tmp mounts being full
• Mount propagation issues
• Per-process filesystem limits

First, check the service-specific mount namespace:

# For systemd services
$ systemctl show apache2 | grep PrivateTmp
PrivateTmp=yes

# View the actual mount points
$ ls -l /proc/$(pgrep apache2)/root/tmp

Then verify the actual available space from the service's perspective:

# Create a test script
$ cat <<'EOF' > /usr/local/bin/check-service-space.sh
#!/bin/bash
pid=$(systemctl show --property MainPID $1 | cut -d= -f2)
nsenter -m -t $pid -- df -h /tmp
EOF

# Run it against your service
$ chmod +x /usr/local/bin/check-service-space.sh
$ ./check-service-space.sh apache2

Option 1: Disable Private /tmp (not recommended for security)

# For systemd services
$ systemctl edit apache2
[Service]
PrivateTmp=false

Option 2: Increase tmpfs Size

# Edit the service unit
$ systemctl edit mysql
[Service]
PrivateTmp=yes
Environment="SYSTEMD_TMPFSSIZE=20%"

Option 3: Clean Legacy tmp Files

# Find and remove old tmp files
$ find /tmp -type f -atime +7 -delete
$ find /var/tmp -type f -atime +7 -delete

# For systemd's private tmp
$ journalctl --disk-usage
$ journalctl --vacuum-size=100M

Add these to your monitoring scripts to catch issues early:

#!/bin/bash
# Check all service private tmp spaces
for service in $(systemctl list-units --type=service --no-legend | awk '{print $1}'); do
    if systemctl show $service | grep -q 'PrivateTmp=yes'; then
        space=$(./check-service-space.sh $service | awk 'NR==2{print $5}')
        echo "$service: $space used in private /tmp"
    fi
done

Recently while upgrading from Debian 7.7 to 8.1, I encountered a puzzling situation where basic service operations like service apache2 restart or /etc/init.d/mysql restart would fail with Error: No space left on device, despite having ample disk space and inodes available.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       443G   28G  393G   7% /

$ df -i
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda1      29450240 419083 29031157    2% /

At first glance, everything appeared normal:

• Plenty of disk space (393G available)
• More than enough inodes (29M free)
• /tmp directory had correct permissions (drwxrwxrwt)
• Could create/delete files in /tmp without issues

The key insight came when examining mount options:

$ mount | grep /dev/sda1
/dev/sda1 on / type ext4 (rw,noexec,nosuid)

Notice the noexec flag? This was preventing execution of temporary files created during service restarts.

To test this theory, I remounted with exec permissions:

# mount -o remount,exec /dev/sda1

After this change, service restarts worked perfectly. For a permanent fix, I updated /etc/fstab:

# Original line
UUID=xxxx / ext4 defaults,noexec,nosuid 0 1

# Modified line
UUID=xxxx / ext4 defaults 0 1

For systems where removing noexec isn't desirable, consider creating a private /tmp for services:

# Create private tmp directory
mkdir /var/lib/mysql/tmp
chown mysql:mysql /var/lib/mysql/tmp

# Add to MySQL configuration
[mysqld]
tmpdir = /var/lib/mysql/tmp

After making changes, verify with:

# Check mount options
$ mount | grep /dev/sda1

# Test service restarts
$ service apache2 restart
$ systemctl restart mysql

• Always check mount options after major OS upgrades
• Consider using systemd-tmpfiles for managing temporary directories
• Monitor dmesg for filesystem-related warnings