Troubleshooting Systemd Environment Variables: Why EnvironmentFile and Direct Definitions Fail

When dealing with systemd services in CentOS 7.3 (or any modern Linux distribution), environment variable management can behave unexpectedly. Here's what's happening under the hood:

# Common pitfall - this looks correct but fails
[Service]
EnvironmentFile=/etc/lc.sh
ExecStart=/usr/bin/env python /opt/app/__init__.py

Systemd handles environment variables differently than shell environments. Here are three verified approaches:

Method 1: Proper EnvironmentFile Syntax

[Service]
# Note the dash (-) which makes the file optional
EnvironmentFile=-/etc/lc.sh
# Critical: Use direct variable expansion in ExecStart
ExecStart=/usr/bin/python /opt/app/__init__.py

Method 2: Inline Environment Variables

[Service]
Environment="LCSQLH=localhost"
Environment="LCSQLU=application"
# Important: Avoid /usr/bin/env when using systemd environments
ExecStart=/usr/bin/python /opt/app/__init__.py

Method 3: Drop-in Directory Configuration

Create /etc/systemd/system/yourservice.service.d/override.conf:

[Service]
Environment="LCSQLH=localhost"
Environment="LCSQLU=application"

1. The /usr/bin/env Problem: Using /usr/bin/env in ExecStart often breaks environment variable inheritance. Systemd provides its own environment handling.

2. File Permissions: Ensure your environment file has proper permissions (usually 640) and is owned by root:root.

3. Variable Expansion Timing: Systemd expands variables at service start, not when reading the unit file. Use systemctl show to verify:

systemctl show yourservice --property=Environment

To verify your variables are actually being passed:

# Check the final environment
systemctl show yourservice -p Environment

# Test with a debug ExecStart
ExecStart=/bin/sh -c 'echo $LCSQLH; sleep 60'

Here's a complete working example for a Python application:

[Unit]
Description=Python Application Service

[Service]
# Load from file (optional)
EnvironmentFile=-/etc/yourapp/env.conf
# Hardcoded fallbacks
Environment="DB_HOST=localhost"
Environment="DB_USER=appuser"

WorkingDirectory=/opt/yourapp
ExecStart=/usr/bin/python3 /opt/yourapp/main.py
Restart=always
User=appuser
Group=appuser

[Install]
WantedBy=multi-user.target

Remember to reload systemd after changes:

systemctl daemon-reload
systemctl restart yourservice

After migrating configuration from hardcoded values to environment variables for my Python application's dev/prod environments, I hit a wall with systemd's environment handling. Despite multiple documented approaches, the variables simply wouldn't propagate to my service process.

The service unit appeared properly configured:

[Unit]
Description=My Application Service
After=network.target

[Service]
EnvironmentFile=/etc/lc.sh
ExecStart=/usr/bin/env python /opt/app/__init__.py
User=appuser
Group=appgroup

Yet the Python code would throw KeyError when accessing os.environ['LCSQLU'], despite the variable being clearly defined in /etc/lc.sh.

Systemd's default security settings were the root cause. Modern systemd versions (particularly on CentOS/RHEL 7+) enable these protections by default:

# Check active service protections
systemd-analyze security myapp.service

The output revealed ProtectSystem=strict and PrivateTmp=yes were isolating the environment.

Option 1: Relax Security Controls (Dev Environment)

Create an override with:

sudo systemctl edit myapp.service

[Service]
ProtectSystem=false
PrivateTmp=false
ReadWritePaths=/etc/lc.sh

Option 2: Proper Environment Injection (Production)

Modify the unit to explicitly pass variables:

[Service]
Environment="LCSQLH=localhost"
Environment="LCSQLU=application"
EnvironmentFile=-/etc/conf.d/myapp
ExecStart=/usr/bin/python /opt/app/__init__.py

Combining both methods with proper permissions:

# /etc/systemd/system/myapp.service.d/override.conf
[Service]
PermissionsStartOnly=true
EnvironmentFile=/etc/lc.sh
ReadWritePaths=/etc/lc.sh
ExecStartPre=/bin/bash -c '. /etc/lc.sh && echo "DB_HOST=$LCSQLH" > /tmp/envdump'

To confirm environment availability:

# Check service's actual environment
sudo systemctl show myapp.service -p Environment
sudo grep -z '' /proc/$(pidof myapp)/environ

# Debug exec line
ExecStart=/bin/sh -c 'env > /tmp/service-env; exec python /opt/app/__init__.py'

CentOS 7's systemd 219 has stricter defaults than earlier versions
EnvironmentFile paths must be explicitly allowed via ReadWritePaths
Combining Environment and EnvironmentFile declarations often works best
Always verify with systemctl show and procfs inspection

ServerDevWorker