Troubleshooting Systemd Environment Variables: Why EnvironmentFile and Direct Definitions Fail


2 views

When dealing with systemd services in CentOS 7.3 (or any modern Linux distribution), environment variable management can behave unexpectedly. Here's what's happening under the hood:

# Common pitfall - this looks correct but fails
[Service]
EnvironmentFile=/etc/lc.sh
ExecStart=/usr/bin/env python /opt/app/__init__.py

Systemd handles environment variables differently than shell environments. Here are three verified approaches:

Method 1: Proper EnvironmentFile Syntax

[Service]
# Note the dash (-) which makes the file optional
EnvironmentFile=-/etc/lc.sh
# Critical: Use direct variable expansion in ExecStart
ExecStart=/usr/bin/python /opt/app/__init__.py

Method 2: Inline Environment Variables

[Service]
Environment="LCSQLH=localhost"
Environment="LCSQLU=application"
# Important: Avoid /usr/bin/env when using systemd environments
ExecStart=/usr/bin/python /opt/app/__init__.py

Method 3: Drop-in Directory Configuration

Create /etc/systemd/system/yourservice.service.d/override.conf:

[Service]
Environment="LCSQLH=localhost"
Environment="LCSQLU=application"

1. The /usr/bin/env Problem: Using /usr/bin/env in ExecStart often breaks environment variable inheritance. Systemd provides its own environment handling.

2. File Permissions: Ensure your environment file has proper permissions (usually 640) and is owned by root:root.

3. Variable Expansion Timing: Systemd expands variables at service start, not when reading the unit file. Use systemctl show to verify:

systemctl show yourservice --property=Environment

To verify your variables are actually being passed:

# Check the final environment
systemctl show yourservice -p Environment

# Test with a debug ExecStart
ExecStart=/bin/sh -c 'echo $LCSQLH; sleep 60'

Here's a complete working example for a Python application:

[Unit]
Description=Python Application Service

[Service]
# Load from file (optional)
EnvironmentFile=-/etc/yourapp/env.conf
# Hardcoded fallbacks
Environment="DB_HOST=localhost"
Environment="DB_USER=appuser"

WorkingDirectory=/opt/yourapp
ExecStart=/usr/bin/python3 /opt/yourapp/main.py
Restart=always
User=appuser
Group=appuser

[Install]
WantedBy=multi-user.target

Remember to reload systemd after changes:

systemctl daemon-reload
systemctl restart yourservice

After migrating configuration from hardcoded values to environment variables for my Python application's dev/prod environments, I hit a wall with systemd's environment handling. Despite multiple documented approaches, the variables simply wouldn't propagate to my service process.

The service unit appeared properly configured:

[Unit]
Description=My Application Service
After=network.target

[Service]
EnvironmentFile=/etc/lc.sh
ExecStart=/usr/bin/env python /opt/app/__init__.py
User=appuser
Group=appgroup

Yet the Python code would throw KeyError when accessing os.environ['LCSQLU'], despite the variable being clearly defined in /etc/lc.sh.

Systemd's default security settings were the root cause. Modern systemd versions (particularly on CentOS/RHEL 7+) enable these protections by default:

# Check active service protections
systemd-analyze security myapp.service

The output revealed ProtectSystem=strict and PrivateTmp=yes were isolating the environment.

Option 1: Relax Security Controls (Dev Environment)

Create an override with:

sudo systemctl edit myapp.service

[Service]
ProtectSystem=false
PrivateTmp=false
ReadWritePaths=/etc/lc.sh

Option 2: Proper Environment Injection (Production)

Modify the unit to explicitly pass variables:

[Service]
Environment="LCSQLH=localhost"
Environment="LCSQLU=application"
EnvironmentFile=-/etc/conf.d/myapp
ExecStart=/usr/bin/python /opt/app/__init__.py

Combining both methods with proper permissions:

# /etc/systemd/system/myapp.service.d/override.conf
[Service]
PermissionsStartOnly=true
EnvironmentFile=/etc/lc.sh
ReadWritePaths=/etc/lc.sh
ExecStartPre=/bin/bash -c '. /etc/lc.sh && echo "DB_HOST=$LCSQLH" > /tmp/envdump'

To confirm environment availability:

# Check service's actual environment
sudo systemctl show myapp.service -p Environment
sudo grep -z '' /proc/$(pidof myapp)/environ

# Debug exec line
ExecStart=/bin/sh -c 'env > /tmp/service-env; exec python /opt/app/__init__.py'
  • CentOS 7's systemd 219 has stricter defaults than earlier versions
  • EnvironmentFile paths must be explicitly allowed via ReadWritePaths
  • Combining Environment and EnvironmentFile declarations often works best
  • Always verify with systemctl show and procfs inspection