How to Permanently Increase Open Files Limit for Long-Running Processes on Debian


2 views

When managing long-running processes (like databases or web servers) on Debian, you might encounter the dreaded "Too many open files" error. This happens when your process hits the system's file descriptor limit. While ulimit -n 2048 provides a temporary solution, we need a permanent configuration that persists across reboots.

Debian handles file limits through two layers:

1. Kernel-level limits (/proc/sys/fs/file-max)
2. User-level limits (set via pam_limits)

Method 1: Editing limits.conf

The most reliable approach is modifying /etc/security/limits.conf:

# Add these lines at the end of the file
*               soft    nofile          2048
*               hard    nofile          4096
yourusername    soft    nofile          8192
yourusername    hard    nofile          16384

Method 2: Systemd Service Configuration

For services managed by systemd, create an override file:

sudo systemctl edit yourservice.service

[Service]
LimitNOFILE=16384

Method 3: Sysctl Configuration

For kernel-level adjustments, edit /etc/sysctl.conf:

fs.file-max = 2097152
fs.nr_open = 2097152

After making changes, verify with:

# For current session
ulimit -n

# For system-wide max
cat /proc/sys/fs/file-max

# For specific process
cat /proc/<pid>/limits | grep "Max open files"

For high-traffic web servers, you might need:

# In /etc/nginx/nginx.conf
worker_rlimit_nofile 40000;

# In systemd override
[Service]
LimitNOFILE=50000
  • Changes require logout/login or reboot to take effect
  • Setting values too high may consume excessive kernel memory
  • Monitor actual usage with lsof | wc -l

When dealing with long-running processes on Debian systems, you might encounter the frustrating "Too many open files" error. This occurs when a process exceeds its file descriptor limit, which by default is often set to a conservative 1024. Let's explore how to make permanent adjustments to overcome this limitation.

First, verify your current limits with:

ulimit -a
# Or specifically for open files:
ulimit -n

The output typically shows:

open files                      (-n) 1024

While ulimit -n 2048 works temporarily, we need persistent configurations. There are three levels to configure:

  1. System-wide limits (affects all users)
  2. User-specific limits
  3. Service/application-specific limits

Edit the system limits configuration file:

sudo nano /etc/security/limits.conf

Add these lines at the end:

* soft nofile 2048
* hard nofile 8192
root soft nofile 8192
root hard nofile 16384

For systemd services, create or edit a service override:

sudo systemctl edit your-service-name

Add these directives:

[Service]
LimitNOFILE=8192
LimitNPROC=4096

After making changes, check with:

cat /proc/$(pidof your-process)/limits | grep "Max open files"

Or for systemd services:

systemctl show your-service-name | grep LimitNOFILE
  • After modifying limits.conf, users must log out and back in
  • For systemd services, run sudo systemctl daemon-reload
  • Check kernel maximum with cat /proc/sys/fs/file-max

For a Node.js application running as a service, your configuration might look like:

# /etc/systemd/system/nodeapp.service
[Unit]
Description=Node.js Application

[Service]
ExecStart=/usr/bin/node /opt/app/server.js
Restart=always
User=nodeuser
Group=nodegroup
Environment=NODE_ENV=production
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target