When working with systemd, you might encounter a frustrating situation where deleted unit files still appear in systemctl list-units output with "not-found" status. These ghost units typically occur when:

systemctl list-units --all firehose-router*
UNIT                       LOAD      ACTIVE SUB    DESCRIPTION
● firehose-router@02.service not-found failed failed firehose-router@02.service
● firehose-router@03.service not-found failed failed firehose-router@03.service

systemd maintains runtime information about units in several locations:

• /run/systemd/transient/ - Temporary runtime units
• /run/systemd/generator/ - Generated units
• /var/lib/systemd/ - Persistent state information

Follow these steps to fully remove lingering units:

# First, stop and disable any active instances
sudo systemctl stop firehose-router@02.service firehose-router@03.service
sudo systemctl disable firehose-router@02.service firehose-router@03.service

# Reset failed state if needed
sudo systemctl reset-failed firehose-router@02.service firehose-router@03.service

# Clean up runtime directories
sudo rm -f /run/systemd/system/firehose-router*
sudo rm -f /run/systemd/transient/firehose-router*
sudo rm -f /var/lib/systemd/{status,linger}/firehose-router*

# Finally, reload systemd
sudo systemctl daemon-reload

After performing these steps, verify the units are gone:

systemctl list-units --all firehose-router*
systemctl show firehose-router@02.service | grep LoadState
journalctl -u firehose-router@02.service --no-pager

To avoid this situation in the future:

1. Always use systemctl disable before deleting unit files
2. Consider using RuntimeDirectory= in unit files for temporary services
3. Clean up template instances properly with systemctl stop 'service@*'

For stubborn cases, examine systemd's internal state:

# Check journal for unit references
journalctl -b -u firehose-router@02.service

# Dump systemd state (requires debug symbols)
sudo gdb -p 1 -batch -ex "print *unit_hashmap" -ex "quit"

When working with systemd, you might encounter situations where units continue to appear in systemctl list-units output even after their corresponding unit files have been deleted. These "ghost units" typically show as "not-found" in the LOAD column but remain visible in the system.

Systemd maintains runtime information about units in several locations:

# Common locations where systemd stores runtime unit information
/run/systemd/system/
/run/systemd/transient/
/run/systemd/generator/

Even when the original unit files are gone, systemd might keep references to these units in its runtime state.

Here's the definitive way to remove these lingering units:

# First, stop and reset the failed units
sudo systemctl stop firehose-router@02.service firehose-router@03.service
sudo systemctl reset-failed firehose-router@02.service firehose-router@03.service

# Then perform a full cleanup
sudo systemctl daemon-reload
sudo systemctl reset-failed  # This resets all failed units

# Verify they're gone
systemctl list-units --all firehose-router*

If the units still persist, try these advanced steps:

# Check for runtime configurations
sudo find /run/systemd/ -name "*firehose-router*"

# Manually remove runtime references
sudo rm -f /run/systemd/system/firehose-router@*.service
sudo rm -f /run/systemd/transient/firehose-router@*.service

# Restart systemd manager
sudo systemctl daemon-reexec

To avoid this situation when removing services:

# Proper removal sequence
sudo systemctl stop service-name
sudo systemctl disable service-name
sudo rm /etc/systemd/system/service-name.service
sudo systemctl daemon-reload
sudo systemctl reset-failed

Remember that template units (@ notation) can be particularly persistent because systemd maintains instances for each template invocation.

For deeper investigation:

# Check unit dependencies
systemd-analyze verify firehose-router@02.service

# Dump the current state
systemd-analyze dump | grep firehose-router