When attempting to increase the open file descriptor limit system-wide on Ubuntu, many administrators find their changes to /etc/security/limits.conf aren't being reflected in ulimit -n output. This persistent 1024 default value indicates the system isn't properly loading the configured limits.

Before diving deeper, verify these essential configurations:

# Confirm limits.conf entries
cat /etc/security/limits.conf | grep nofile

# Check PAM module inclusion
grep -r "pam_limits" /etc/pam.d/

# Verify system-wide maximum
cat /proc/sys/fs/file-max

1. Systemd Service Limitations: Modern Ubuntu systems using systemd often ignore limits.conf for services. Create override files:

# For sshd service
sudo mkdir -p /etc/systemd/system/sshd.service.d/
sudo tee /etc/systemd/system/sshd.service.d/override.conf << EOF
[Service]
LimitNOFILE=100000
EOF

2. User Session Scope: Interactive login sessions may require additional configuration in /etc/systemd/logind.conf:

# Edit or add these lines
DefaultLimitNOFILE=100000
DefaultLimitNOFILESoft=100000

After making changes, perform these verification steps:

# Reload systemd configurations
sudo systemctl daemon-reload
sudo systemctl restart sshd

# Check current session limits
ulimit -n

# Check another method
cat /proc/$$/limits | grep "Max open files"

For permanent system-wide changes, modify /etc/sysctl.conf:

# Add these lines
fs.file-max = 762659
fs.nr_open = 1000000

# Apply changes
sudo sysctl -p

When all else fails, trace the limit initialization process:

strace -e trace=open,openat,access -f /bin/bash -c 'ulimit -n' 2>&1 | grep limits

This helps identify if and when the system attempts to read the limits configuration files.

---

After configuring /etc/security/limits.conf with increased file descriptor limits and verifying pam_limits.so inclusion across relevant PAM modules, you'd expect ulimit -n to reflect your changes. Yet the system stubbornly clings to the default 1024 limit. Let's systematically investigate why this happens.

First, confirm whether the limits are actually being read by the system:

# Check current session limits
cat /proc/$$/limits | grep "Max open files"

# Alternative method
grep -i "max open files" /proc/$$/limits

Modern Ubuntu systems using systemd require additional configuration:

# Check if systemd is ignoring PAM limits
systemctl show --property DefaultLimitNOFILE

# Permanent solution:
sudo nano /etc/systemd/system.conf
# Add or modify:
DefaultLimitNOFILE=100000

Limits apply to new login sessions. Verify if you're testing in the same session where changes were made:

# Start a fresh login session
sudo -i
ulimit -n

Remote sessions might have additional restrictions:

# Check SSH daemon configuration
grep -i "MaxSessions\|MaxStartups" /etc/ssh/sshd_config

# Add these to /etc/ssh/sshd_config if missing:
MaxSessions 100
MaxStartups 100

Some applications set their own limits. Check for:

# Common locations for service-specific limits
ls -l /etc/systemd/system/*.service
grep -r "LimitNOFILE" /etc/systemd/system/

While file-max looks sufficient, check related parameters:

# View all relevant kernel parameters
sysctl -a | grep -E "file|inode|dentry"

# Important parameters to check:
cat /proc/sys/fs/nr_open
cat /proc/sys/fs/file-nr

Enable PAM debugging to trace limit application:

# Temporarily modify PAM configuration
sudo nano /etc/pam.d/common-session
# Change to:
session required pam_limits.so debug

# Then monitor auth logs:
tail -f /var/log/auth.log

For persistent issues, consider:

# Create a custom limits.d file
echo "* - nofile 100000" | sudo tee /etc/security/limits.d/99-custom.conf

# Or set via profile:
echo "ulimit -n 100000" | sudo tee /etc/profile.d/custom_limits.sh

Before concluding:

1. Reboot the system or at least restart affected services
2. Verify in a clean login session (not sudo, but fresh SSH)
3. Check both soft and hard limits (ulimit -Sn and ulimit -Hn)
4. Confirm no SELinux/AppArmor restrictions exist