Debugging systemd “Loaded: not-found” Error When Service File Exists as Symlink


5 views

While working with systemd services, you might encounter a puzzling situation where:

systemctl status your-service.service
● your-service.service
   Loaded: not-found (Reason: No such file or directory)
   Active: active (running)

This occurs despite the service file being properly symlinked in /etc/systemd/system. Let's examine why this happens and how to resolve it.

The primary culprit is often the dynamic nature of symlinks and how systemd caches service information. When the actual service file resides on a separate mount point (like /opt in your case), systemd might lose track of it if:

  • The mount becomes temporarily unavailable
  • The symlink target changes after systemd initialization
  • The filesystem isn't mounted when systemd scans for services

1. The Immediate Fix: Daemon Reload

sudo systemctl daemon-reload

2. Permanent Solution: Proper Mount Dependencies

Ensure your service file is always available by adding mount dependencies to the service unit:

[Unit]
After=network.target remote-fs.target
Requires=remote-fs.target

3. Alternative Approach: Copy Instead of Symlink

For critical services, consider copying rather than symlinking:

sudo cp /opt/app/daps-prod/resource-service/bin/resource-service-prod-prod.service \
        /etc/systemd/system/resource-service-prod.service

To troubleshoot further, use these commands:

# Check symlink resolution
readlink -f /etc/systemd/system/resource-service-prod.service

# Verify file existence
ls -la /opt/app/daps-prod/resource-service/bin/resource-service-prod-prod.service

# Check mount status
mount | grep /opt
  • Place frequently accessed services directly in /etc/systemd/system
  • For development environments, use WantedBy=multi-user.target
  • Include proper [Install] section for service persistence
  • Set correct file permissions: chmod 644 /etc/systemd/system/*.service

Here's a properly configured example for reference:

[Unit]
Description=Resource Service
After=network.target remote-fs.target
Requires=remote-fs.target

[Service]
Type=simple
User=appuser
WorkingDirectory=/opt/app/daps-prod/resource-service
ExecStart=/usr/bin/node /opt/app/daps-prod/resource-service/source/app.js
Restart=always

[Install]
WantedBy=multi-user.target

Remember to run systemctl daemon-reload after any changes to service files.

I recently encountered a puzzling situation where systemctl reported a service as "not-found" despite the .service file being present as a symlink. The key details:

ls -lrt /etc/systemd/system
lrwxrwxrwx 1 root root 96 Mar 5 2017 resource-service-prod.service -> /opt/app/daps-prod/resource-service/bin/resource-service-prod-prod.service

Yet systemctl status showed:

● resource-service-prod-prod.service
   Loaded: not-found (Reason: No such file or directory)
   Active: active (running)

The service would stop normally but fail to restart, requiring a systemctl daemon-reload to work again. This occurred because:

  • The actual service file resided on a separate mount
  • Systemd temporarily lost track of the symlink target
  • Mount timing issues could cause detection problems

After extensive testing, I identified these contributing factors:

# Check if the symlink resolves
readlink -f /etc/systemd/system/resource-service-prod.service

# Verify the target exists
ls -la /opt/app/daps-prod/resource-service/bin/resource-service-prod-prod.service

# Check mount status
mount | grep /opt/app/daps-prod

The fundamental issue was that systemd's unit file cache wasn't properly tracking symlinks to files on separate filesystems.

Here are three effective workarounds:

  1. Use absolute paths in symlinks: 
    sudo ln -sf /absolute/path/to/service/file /etc/systemd/system/
  2. Add dependency on the mount: 
    [Unit]
After=mount-point.mount
Requires=mount-point.mount
  3. Create a local copy instead of symlink: 
    sudo cp /original/path/service-file.service /etc/systemd/system/
sudo systemctl daemon-reload

To avoid similar issues:

  • Regularly check symlink validity with sudo systemctl list-unit-files --state=bad
  • Implement a pre-start script to verify paths:
[Service]
ExecStartPre=/bin/bash -c 'test -f /path/to/service || exit 1'

This ensures the service won't start if critical files are unavailable.

Essential troubleshooting tools:

# Show all loaded units
systemctl list-units

# Check symlink resolution
systemd-analyze verify /etc/systemd/system/your-service.service

# Detailed unit inspection
systemctl show your-service.service

# Follow journal logs
journalctl -u your-service.service -f

Remember that systemd caches unit information, so changes require either a daemon-reload or system restart to take effect.