When working with complex systemd unit files that utilize multiple environment variables in ExecStart or ExecStop directives, troubleshooting becomes particularly challenging. The expansion happens internally within systemd and isn't typically visible in logs or status outputs.

Here are several effective methods to inspect the fully expanded command:

Method 1: Temporary Echo Command Substitution

Modify your unit file temporarily to output the command instead of executing it:

[Service]
ExecStart=/bin/echo "RESULT: /usr/bin/java $OPTS_COMMON $OPTS $OPTS_LOG $OPTS_DEBUG some.class.Start --param1 ${PARAM1} --param2 ${PARAM2}"

Then check the journal:

journalctl -u yourservice.service -b

Method 2: Using systemd-analyze

For a dry-run analysis:

systemd-analyze verify yourservice.service

While this won't show the expanded command, it helps validate syntax and environment loading.

Method 3: Environment Inspection

Create a helper script to log the environment:

[Service]
ExecStart=/bin/bash -c 'env > /tmp/service-env.log; exec /usr/bin/java $OPTS_COMMON $OPTS $OPTS_LOG $OPTS_DEBUG some.class.Start --param1 ${PARAM1} --param2 ${PARAM2}'

Using systemd.exec Man Page References

Always consult:

man systemd.exec

Particularly the sections about command line parsing and environment variable handling.

Example Unit File with Debugging

Here's a comprehensive example demonstrating variable expansion inspection:

[Unit]
Description=Java Service with Debugging
After=network.target

[Service]
EnvironmentFile=/etc/default/javaservice
ExecStartPre=/bin/sh -c 'echo "Final command would be: /usr/bin/java $OPTS_COMMON $OPTS $OPTS_LOG $OPTS_DEBUG some.class.Start --param1 ${PARAM1} --param2 ${PARAM2}" > /tmp/java-service-cmd.debug'
ExecStart=/usr/bin/java $OPTS_COMMON $OPTS $OPTS_LOG $OPTS_DEBUG some.class.Start --param1 ${PARAM1} --param2 ${PARAM2}
Restart=on-failure

[Install]
WantedBy=multi-user.target

• Always document environment variables in comments
• Consider using separate EnvironmentFile for easier maintenance
• Implement logging of critical variables during service startup
• Use systemd-cat to capture debug output directly in the journal

---

When working with complex systemd unit files containing multiple environment variables in ExecStart directives, debugging becomes challenging because systemd doesn't natively show the fully expanded command line with all variables substituted. Consider this common scenario:

[Service]
Environment=OPTS_COMMON="-Xmx1G -Xms512M"
Environment=OPTS="-Dconfig.file=/etc/app/config.conf"
Environment=OPTS_DEBUG="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n"
ExecStart=/usr/bin/java $OPTS_COMMON $OPTS $OPTS_DEBUG some.class.Start

Method 1: Temporary Echo Replacement

The most straightforward way to see the expanded command:

ExecStart=/bin/echo "Resulting command:" /usr/bin/java $OPTS_COMMON $OPTS $OPTS_DEBUG some.class.Start

Remember to:

1. Set Type=oneshot in [Service] section
2. Disable or modify any ExecStop commands
3. Run systemctl daemon-reload before testing

Method 2: systemd-analyze Verification

While not showing expanded commands, this helps validate unit syntax:

systemd-analyze verify your-service.service

Method 3: Environment Dumping

Create a helper script to log the complete environment:

#!/bin/bash
env > /tmp/service-environment.log
exec /usr/bin/java "$@"

Then modify your unit:

ExecStart=/path/to/helper_script.sh $OPTS_COMMON $OPTS $OPTS_DEBUG some.class.Start

Using systemd.exec Man Page References

Many expansion cases are documented in:

man systemd.exec

Particularly note the sections about:

• Command line syntax
• Environment variable substitution
• Specifier expansion (%i, %n, etc.)

Dynamic Debugging with systemd-run

For transient services, you can test expansions without permanent changes:

systemd-run --unit=test-expansion \
  --property=Environment="OPTS_COMMON=-Xmx1G" \
  --property=ExecStart="/bin/echo /usr/bin/java $OPTS_COMMON some.class.Start" \
  --property=Type=oneshot

[Service]
# Store environment in separate file
EnvironmentFile=/etc/default/myapp

# Use explicit paths
ExecStart=/usr/bin/bash -c 'echo "Full command: /usr/bin/java ${OPTS} some.class.Start" && \
  exec /usr/bin/java ${OPTS} some.class.Start'

# Document variable purposes
# OPTS: JVM runtime options
# OPTS_DEBUG: Remote debugging parameters

For production systems where modifying units isn't desirable:

1. Implement a wrapper script with extensive logging
2. Use systemctl show to inspect all properties
3. Leverage journalctl -u your-service -o verbose for detailed logs