When debugging systemd oneshot services (particularly those triggered by timers), a common pain point emerges: journalctl's --unit=foo.service shows all executions concatenated together. This becomes problematic when you need to isolate logs from just the last invocation.

Systemd provides several metadata fields that can help isolate specific runs:

# Show all available fields for a unit
journalctl -u foo.service -o json-pretty | jq '.[0] | keys'

The most robust approach uses _SYSTEMD_INVOCATION_ID - a unique identifier generated for each service invocation:

# Get the invocation ID of the last run
LAST_INVOCATION=$(systemctl show foo.service --property=InvocationID --value)

# Filter logs using the ID
journalctl _SYSTEMD_INVOCATION_ID=$LAST_INVOCATION

For systems without _SYSTEMD_INVOCATION_ID (pre v230), consider these workarounds:

# 1. Using boot ID and timestamp
journalctl -u foo.service --since "2023-05-01 14:00:00" --until "2023-05-01 14:05:00"

# 2. Combining with process information
journalctl -u foo.service _PID=$(systemctl show foo.service --property=ExecMainPID --value)

For recurring debugging needs, add this to your oneshot service:

[Service]
ExecStartPost=/bin/sh -c 'echo "InvocationID: $INVOCATION_ID" > /tmp/foo-last-run'

Combine multiple filters for precision:

# Get logs between two specific invocations
journalctl _SYSTEMD_INVOCATION_ID=abc123..def456

# Show logs with priority 3 (error) or higher
journalctl _SYSTEMD_INVOCATION_ID=$LAST_INVOCATION -p 3

When debugging systemd services (especially oneshot services triggered by timers), developers often need to isolate logs from a specific service instance. The standard journalctl --unit=foo.service approach falls short because it aggregates all executions without distinguishing between individual runs.

While PID-based filtering seems logical:

journalctl _PID=12345

This method has three critical flaws:

• PID reuse between service runs
• Services that fork multiple processes
• Difficulty in programmatically retrieving the previous run's PID

systemd actually provides better identifiers through metadata fields:

# Filter by invocation ID (best modern solution)
journalctl _SYSTEMD_INVOCATION_ID=$(systemctl show -p InvocationID foo.service --value)

# Alternative: Filter by control group
journalctl _SYSTEMD_CGROUP=$(systemctl show -p ControlGroup foo.service --value)

For a timer-triggered oneshot service:

# 1. Find the latest invocation
INVOCATION_ID=$(journalctl --unit=foo.service -n 1 --output=json | jq -r '._SYSTEMD_INVOCATION_ID')

# 2. Filter logs for that specific run
journalctl _SYSTEMD_INVOCATION_ID=${INVOCATION_ID} --no-pager

Ensure your journald.conf has sufficient retention:

[Journal]
Storage=persistent
SystemMaxUse=1G
MaxRetentionSec=1month

When you need approximate time-based filtering:

# Get approximate start/end times
systemctl show -p ExecMainStartTimestamp foo.service
systemctl show -p ExecMainExitTimestamp foo.service

# Then use time window filtering
journalctl --unit=foo.service --since "2023-06-01 14:00:00" --until "2023-06-01 14:01:00"

For regular debugging, create a bash function:

function service-logs() {
    local unit=$1
    local invocation_id=$(journalctl --unit="$unit" -n 1 --output=json | jq -r '._SYSTEMD_INVOCATION_ID')
    journalctl _SYSTEMD_INVOCATION_ID="$invocation_id" --no-pager
}