Persistent sysctl Settings Not Applied on Reboot: Debugging and Fixing Kernel Parameter Issues


6 views

After multiple reboots on my Lubuntu 18.10 system, I discovered that certain critical sysctl parameters weren't persisting despite being properly configured in various configuration files. The most concerning was net.ipv4.conf.all.rp_filter=1 which kept reverting to 0, potentially exposing the system to IP spoofing attacks.

First, I mapped all potential configuration locations using:

find / -name '*sysctl*.conf'

The relevant files found were:

  • /etc/sysctl.conf
  • /etc/ufw/sysctl.conf
  • /etc/sysctl.d/99-sysctl.conf

Through testing, I discovered that Wireguard VPN was automatically overriding rp_filter settings. According to Ubuntu's documentation, this is expected behavior to prevent legitimate VPN packets from being dropped.

For systems requiring strict reverse path filtering, you can add this to your Wireguard configuration:

PostUp = sysctl -w net.ipv4.conf.all.rp_filter=2
PostDown = sysctl -w net.ipv4.conf.all.rp_filter=1

The martian packet logging parameters (log_martians) were being overridden by UFW's configuration. To fix this, either:

  1. Edit /etc/ufw/sysctl.conf directly:
  2. net.ipv4.conf.all.log_martians=1
    net.ipv4.conf.default.log_martians=1
  3. Or create a higher-priority file in /etc/sysctl.d/:
  4. # /etc/sysctl.d/10-network-security.conf
    net.ipv4.conf.all.log_martians=1
    net.ipv4.conf.default.log_martians=1

The fs.suid_dumpable setting proved most stubborn. After research, I found systemd might be resetting this. The permanent solution was:

# Create systemd override
mkdir -p /etc/systemd/system.conf.d
echo "[Manager]" > /etc/systemd/system.conf.d/suid_dumpable.conf
echo "DefaultLimitSUID=0" >> /etc/systemd/system.conf.d/suid_dumpable.conf

To automatically verify settings after boot, I created this cron job:

#!/bin/bash
EXPECTED=(
  "fs.suid_dumpable = 0"
  "net.ipv4.conf.all.rp_filter = 1" 
  "net.ipv4.conf.all.log_martians = 1"
)

for setting in "${EXPECTED[@]}"; do
  if ! sysctl -a 2>/dev/null | grep -q "$setting"; then
    logger -t sysctl-check "Failed setting: $setting"
    # Attempt remediation
    sysctl -p /etc/sysctl.d/*.conf
  fi
done

The key takeaways for making sysctl settings persistent:

  • Use /etc/sysctl.d/ with numbered prefixes (e.g., 10-, 20-) for proper load ordering
  • Check for application-specific overrides (like UFW or VPNs)
  • Verify with sysctl --system to test all configuration files
  • Consider systemd services that might reset parameters

On Lubuntu 18.10 (Cosmic Cuttlefish), certain sysctl parameters fail to persist after reboot despite being properly configured. The most critical one is net.ipv4.conf.all.rp_filter=1, which protects against IP spoofing attacks but gets reset to 0 during boot.

Through testing with Lynis and manual verification, we identified four problematic parameters:

# Current non-persistent settings
fs.suid_dumpable=0
net.ipv4.conf.all.rp_filter=1  # Overridden by WireGuard
net.ipv4.conf.all.log_martians=1  # Overridden by UFW
net.ipv4.conf.default.log_martians=1  # Overridden by UFW

WireGuard VPN intentionally disables rp_filter (setting it to 0) to prevent legitimate packet drops, as documented in Ubuntu's ImprovedNetworking documentation. This creates a security versus functionality trade-off.

The system loads sysctl settings from multiple locations, with later files overriding earlier ones:

find / -name '*sysctl*.conf' 2>/dev/null
/etc/sysctl.conf
/etc/ufw/sysctl.conf  # UFW overrides settings here
/etc/sysctl.d/99-sysctl.conf  # Final override point

Option 1: Create a higher-priority config file

# Create new config file with higher priority than UFW/WireGuard
sudo tee /etc/sysctl.d/90-custom.conf <

Option 2: Modify UFW's sysctl configuration

# Edit UFW's override file
sudo nano /etc/ufw/sysctl.conf

# Comment out or modify conflicting lines:
# net/ipv4/conf/all/log_martians=0
# net/ipv4/conf/default/log_martians=0

# Then restart UFW
sudo systemctl restart ufw

For systems using WireGuard, consider this post-up script solution:

# Create WireGuard post-up script
sudo tee /etc/wireguard/scripts/post-up.sh <

After implementing changes, verify persistence:

# Check current values
sysctl fs.suid_dumpable \
      net.ipv4.conf.all.rp_filter \
      net.ipv4.conf.all.log_martians \
      net.ipv4.conf.default.log_martians

# Reboot and verify again
sudo reboot
  • Check boot sequence with dmesg | grep sysctl
  • View loading order with systemd-analyze cat-config sysctl.d
  • Test individual files with sysctl -p /path/to/file.conf