Debugging Missing stdout/stderr Logs in systemd Services: Complete Guide to Log Capture Methods


2 views

When running services under systemd, many developers encounter the frustrating situation where their application's stdout/stderr output disappears into the void. This happens because systemd handles process output differently than direct terminal execution.

The most systemd-native approach is using journald. First, verify journald is actually capturing your logs:

# Check if journald shows any output at all
journalctl -u your-service.service -f

If nothing appears, your service might be:

  • Buffering output
  • Using direct file logging instead of stdout
  • Running with output redirected elsewhere

Add these directives to your service file:

[Service]
StandardOutput=journal
StandardError=journal

Your attempt with file output was close. Here's the working version:

[Service]
StandardOutput=file:/var/log/your-service/out.log
StandardError=file:/var/log/your-service/err.log

Critical notes:

  • Ensure the directory exists and is writable
  • The service must run as a user with write permissions
  • Consider log rotation for production systems

When files appear but stay empty:

# Check file permissions
ls -la /var/log/your-service/

# Test manual write
sudo -u your-service-user echo "test" >> /var/log/your-service/out.log

# Verify SELinux/AppArmor isn't blocking
sudo ausearch -m avc -ts recent

For comprehensive logging:

[Service]
StandardOutput=journal+file:/var/log/your-service/out.log
StandardError=journal+file:/var/log/your-service/err.log

Here's a complete working example for a Python service:

[Unit]
Description=My Python Service

[Service]
User=service-user
Group=service-group
WorkingDirectory=/opt/my-service
ExecStart=/usr/bin/python3 /opt/my-service/main.py
Restart=always
Environment=PYTHONUNBUFFERED=1
StandardOutput=journal+file:/var/log/my-service/out.log
StandardError=journal+file:/var/log/my-service/err.log

[Install]
WantedBy=multi-user.target

Key points in this example:

  • PYTHONUNBUFFERED prevents output buffering
  • Explicit user/group ensures proper permissions
  • WorkingDirectory helps with relative paths

When debugging systemd services, one common frustration occurs when the application's stdout/stderr output mysteriously disappears - even though you see these logs when running the service manually. The standard troubleshooting steps like journalctl -u service-name often fail to reveal the actual application logs, showing only service lifecycle events.

The StandardOutput and StandardError directives in your service file should theoretically work, but several factors could be preventing proper logging:


# Common problematic service file example
[Service]
ExecStart=/usr/bin/my-service
StandardOutput=append:/var/log/my-service/stdout.log
StandardError=append:/var/log/my-service/stderr.log

Solution 1: Ensure Proper File Permissions

Create the log directory with correct permissions before starting the service:


sudo mkdir -p /var/log/my-service
sudo chown root:root /var/log/my-service
sudo chmod 755 /var/log/my-service

Solution 2: Use Journald Integration

Modern systemd versions (v236+) have better journald integration:


[Service]
ExecStart=/usr/bin/my-service
StandardOutput=journal
StandardError=journal

Then query with:


journalctl -u my-service -o cat

Solution 3: Force Immediate Flushing

Some applications buffer output. Add this to your service file:


[Service]
Environment="PYTHONUNBUFFERED=1"  # For Python apps
ExecStart=/usr/bin/stdbuf -oL -eL /usr/bin/my-service

Using systemd-analyze

Inspect potential issues with:


systemd-analyze verify /etc/systemd/system/my-service.service

Testing with Temporary Service

Create a test service to verify logging works:


[Unit]
Description=Logging test service

[Service]
Type=simple
ExecStart=/bin/sh -c 'while true; do echo "Test $(date)"; sleep 1; done'
StandardOutput=journal+console
StandardError=journal+console

[Install]
WantedBy=multi-user.target

Here's a comprehensive service file that addresses common issues:


[Unit]
Description=My Application Service
After=network.target

[Service]
Type=simple
User=appuser
Group=appgroup
WorkingDirectory=/opt/my-app
Environment="PYTHONUNBUFFERED=1"
Environment="LOG_LEVEL=DEBUG"
ExecStart=/usr/bin/stdbuf -oL -eL /usr/bin/my-service --daemon
Restart=on-failure
RestartSec=5s
StandardOutput=file:/var/log/my-service/out.log
StandardError=file:/var/log/my-service/err.log
SyslogIdentifier=my-service

[Install]
WantedBy=multi-user.target