I recently encountered an interesting scenario with two CentOS 5 servers that appeared identical in specifications but showed different default process limits (ulimit -u). One returned unlimited while the other showed 77824. This inconsistency persisted even in cron jobs, indicating these weren't shell-specific settings.

After extensive searching, I discovered that default ulimit values in Linux systems can come from several sources:

1. Kernel parameters (check via: cat /proc/sys/kernel/threads-max)
2. PAM modules configuration (/etc/security/limits.conf and /etc/security/limits.d/*)
3. Systemd configuration (for CentOS 7+) in /etc/systemd/system.conf
4. init.d scripts in /etc/init.d/
5. The kernel itself (particularly for nproc limits)

In CentOS 6, I found that the wildcard (*) in /etc/security/limits.conf doesn't work for nproc limits, though it works fine for nofile. The solution was to specify each user individually:

# This works in CentOS 6 for individual users
username1 - nproc unlimited
username2 - nproc unlimited

# But this doesn't work for nproc (though works for nofile)
* - nproc unlimited

For system-wide changes that persist across reboots:

# Option 1: For all users (works in CentOS 5)
echo "* - nproc unlimited" >> /etc/security/limits.conf

# Option 2: For systemd systems (CentOS 7+)
echo "DefaultLimitNPROC=unlimited" >> /etc/systemd/system.conf
systemctl daemon-reload

# Option 3: Temporary kernel parameter change
sysctl -w kernel.threads-max=1000000

To trace where limits are being set:

# Check effective limits for a process
cat /proc/<pid>/limits

# Check PAM configuration
grep -r pam_limits /etc/pam.d/

# Check systemd defaults
systemctl show --property=DefaultLimitNPROC

The differences between identical systems might come from:

• Different kernel versions or patches
• Variations in PAM configuration
• Customizations in /etc/sysctl.conf
• Different installation media or post-install scripts

After encountering inconsistent process limits (ulimit -u) across CentOS 5 and 6 servers, I discovered some surprising behaviors in how Linux handles these limits. Here's what every sysadmin should know about ulimit configuration.

The primary configuration files that affect ulimit values are:

1. /etc/security/limits.conf
2. /etc/security/limits.d/*.conf
3. Systemd service files (for CentOS 7+)
4. PAM modules configuration

In CentOS 5, wildcards work as expected in /etc/security/limits.conf:

* - nproc unlimited  # Works in CentOS 5

But in CentOS 6, you must specify users individually:

username1 - nproc unlimited
username2 - nproc unlimited

The default values come from multiple sources:

• Kernel parameters (check /proc/sys/kernel/threads-max)
• PAM module defaults
• Systemd service defaults (for newer systems)
• Distro-specific compiled defaults

For cron jobs: Set limits directly in the crontab:

* * * * * ulimit -u 50000 && /path/to/command

System-wide configuration: For CentOS 6, modify /etc/security/limits.conf:

# Set for specific users
username soft nproc 65535
username hard nproc 65535

# Or use groups
@developers soft nproc 65535
@developers hard nproc 65535

To debug where limits are being set:

# Check effective limits for a process
cat /proc/[PID]/limits

# Check PAM configuration
grep pam_limits /etc/pam.d/*

# Verify systemd overrides (if applicable)
systemctl show [service] | grep Limit

Remember that changes to limits.conf require a new login session to take effect.

For systemd-based systems (CentOS 7+), create override files:

# /etc/systemd/system/[service].d/override.conf
[Service]
LimitNPROC=65535

For login sessions, ensure proper PAM configuration in /etc/pam.d/login and /etc/pam.d/sshd.