When working with AWS EC2 instances and custom AMIs, one common pain point is service startup sequencing relative to user-data execution. The default Ubuntu 16.04 cloud-init behavior creates a tricky dependency situation where:

cloud-init (cloud-final.service) → User-data scripts → Your custom service

The fundamental issue stems from cloud-final.service using RemainAfterExit=yes, which means it technically never "completes" from systemd's perspective, making traditional After= dependencies ineffective.

Here are three battle-tested approaches to solve this sequencing problem:

1. Using systemd Path Units

Create a path unit that watches for a user-data completion marker:

[Unit]
Description=Watch for user-data completion
Before=your-service.service

[Path]
PathExists=/var/lib/cloud/instance/boot-finished
Unit=your-service.service

[Install]
WantedBy=multi-user.target

2. Custom Startup Script with cloud-init Helper

Leverage cloud-init's native runcmd to trigger your service:

#cloud-config
runcmd:
  - systemctl start your-service

3. Modified Service Unit with ExecStartPre

Add explicit checks in your service definition:

[Unit]
Description=Your Custom Service
After=network.target cloud-init.target

[Service]
Type=simple
ExecStartPre=/bin/sh -c 'while [ ! -f /var/lib/cloud/instance/boot-finished ]; do sleep 2; done'
ExecStart=/usr/bin/your-service
Restart=on-failure

[Install]
WantedBy=multi-user.target

When choosing between these approaches, consider:

• Timeout Handling: The Path Unit method may need additional timeout logic
• Dependencies: Method #3 makes dependencies explicit in the unit file
• Cloud-Init Version: Check /var/lib/cloud/instance/boot-finished exists in your AMI

After implementation, verify sequencing with:

journalctl -u cloud-final -u your-service --no-pager -o cat

For debugging startup timing issues:

systemd-analyze critical-chain your-service.service

---

When working with AWS EC2 instances, many developers encounter a common scenario: they need their custom systemd service to start after the user-data script completes execution. The standard approach of using After=cloud-final.service fails because:

# Check cloud-final.service properties
$ systemctl show cloud-final.service | grep RemainAfterExit
RemainAfterExit=yes

This RemainAfterExit=yes setting means the unit never technically "finishes," preventing dependent services from starting.

Method 1: Using cloud-init modules

The most robust solution is to leverage cloud-init's module system:

# Example /etc/cloud/cloud.cfg.d/99-my-service.cfg
cloud_final_modules:
 - [scripts-user, always]
 - [my-service-startup, once]

Then create a custom cloud-init module:

# /etc/cloud/cloud.cfg.d/my-service-startup.cfg
#cloud-config
runcmd:
 - systemctl start my-custom-service.service

Method 2: Create a custom target

For more control, create a dedicated target:

# /etc/systemd/system/user-data-complete.target
[Unit]
Description=User Data Processing Complete
After=cloud-final.service

Then modify your service:

[Unit]
Description=My Custom Service
After=user-data-complete.target
Requires=user-data-complete.target

[Service]
ExecStart=/usr/bin/my-service
Restart=on-failure

[Install]
WantedBy=multi-user.target

Method 3: Directly trigger from user-data

If you control the user-data script, simply add the service start command at the end:

#!/bin/bash
# Your user-data script content
apt-get update
apt-get install -y some-package
# ...
systemctl enable --now my-custom-service.service

To confirm your service starts after user-data completion:

# Check boot logs
$ journalctl -b | grep -E 'cloud-init|my-custom-service'

# Alternative using systemd-analyze
$ systemd-analyze critical-chain my-custom-service.service

• Check cloud-init logs: /var/log/cloud-init.log
• Verify execution order: systemd-analyze plot > boot-analysis.svg
• Add explicit waits in user-data if needed: cloud-init status --wait