When dealing with Linux server administration, one particularly puzzling scenario occurs when the number of open file descriptors for the root user appears to exceed the system's ulimit -n setting. Let's examine this phenomenon and understand what's really happening.

The ulimit -n command shows the maximum number of file descriptors that can be opened by a single process at any given time. However, there are several nuances to this limit:

# Check current limits
ulimit -n
# Output typically shows 1024 for many systems

# View system-wide limits
cat /proc/sys/fs/file-max
# Shows the absolute system-wide limit

The apparent discrepancy occurs because:

• File descriptors are counted per process, not per user
• The root user can have multiple processes each using descriptors
• Different processes can reference the same file through different descriptors

The initial approach using lsof -u root | wc -l is misleading because:

# This counts all file references, including:
# - Memory mapped files
# - Multiple handles to same file
# - Special file descriptors

# More accurate counting:
lsof -u root -a -d '^mem' | awk '{print $9}' | sort -u | wc -l

Two critical limits exist in Linux:

# 1. Per-process limit (ulimit -n)
# 2. System-wide limit (/proc/sys/fs/file-max)

# To check current usage:
cat /proc/sys/fs/file-nr
# Output format: allocated unused system-max

In the case mentioned with Glassfish showing 1300 descriptors:

# Check process-specific limits
cat /proc/$(pgrep -f glassfish)/limits | grep 'open files'

# Verify actual unique files open
lsof -p $(pgrep -f glassfish) | awk '/\/.*\// {print $9}' | sort -u | wc -l

The "too many open files" error typically appears when:

• A single process hits its ulimit -n threshold
• The system-wide limit is reached (rare for modern servers)
• File descriptor leaks occur in applications

For systems needing higher limits:

# Temporary increase (root):
ulimit -n 65536

# Permanent system-wide changes:
# /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

# And in /etc/sysctl.conf:
fs.file-max = 2097152

Regular monitoring helps prevent issues:

# Watch system-wide usage:
watch -n 5 'cat /proc/sys/fs/file-nr'

# Top processes by FD count:
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head
lsof -p <PID> | wc -l

When troubleshooting file descriptor exhaustion on our Linux server, I encountered a puzzling scenario where lsof -u root | wc -l reported 2500 open file descriptors despite ulimit -n showing a limit of 1024. This apparent contradiction reveals important nuances about how Linux handles file descriptors.

The key realization is that ulimit -n controls the number of unique file descriptions (kernel structures) per process, not the count of file descriptor entries across all processes. Multiple descriptors can reference the same file description, and different processes can share file descriptions through inheritance or IPC.

# To get accurate count of unique open files:
lsof -u root -Fn | awk -F'\0' '/^n/ {print $2}' | sort -u | wc -l

The root user has several special capabilities regarding resource limits:

• Can spawn multiple processes, each with its own FD limit
• Can bypass some restrictions through capabilities (CAP_SYS_RESOURCE)
• May have different limits in different sessions

Here's a more precise way to analyze FD usage across processes:

# Check per-process FD counts
for pid in $(pgrep -u root); do
    echo "PID $pid: $(ls /proc/$pid/fd | wc -l) FDs"
done | sort -nk3

# View system-wide FD usage
cat /proc/sys/fs/file-nr

The ulimit restriction comes into play during new FD creation attempts. You'll hit the limit when:

• A single process tries to open more files than its limit
• The system-wide FD limit is reached (check via cat /proc/sys/fs/file-max)
• Inode cache pressure causes allocation failures

To properly manage FD limits:

# Permanent system-wide configuration
echo "fs.file-max = 500000" >> /etc/sysctl.conf
sysctl -p

# Per-process limits via systemd
[Service]
LimitNOFILE=65536

When facing mysterious FD exhaustion, this script helps identify leaks:

#!/bin/bash
watch -n 5 '
echo "Total FDs: $(lsof | wc -l)"
echo "System FD usage: $(cat /proc/sys/fs/file-nr)"
for p in $(pgrep -f glassfish); do
    echo "PID $p: $(ls /proc/$p/fd | wc -l) FDs"
done
'