Ubuntu 16.04 marked a significant shift from Upstart to systemd as the default init system. This explains why your existing Upstart configuration (/etc/init/crawler.conf) no longer works. Let's adapt your PHP crawler service properly.

Where Upstart used .conf files in /etc/init/, systemd uses .service files in /etc/systemd/system/. The syntax and capabilities also differ substantially.

Create a new file at /etc/systemd/system/crawler.service:

[Unit]
Description=PHP Crawler for URL Processing
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/php -f /var/www/crawler.php
Restart=on-failure
RestartSec=5s
StartLimitInterval=100
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

The Restart directives replace Upstart's respawn functionality. The StartLimit* settings handle the respawn limit equivalent.

After creating the file, run these commands:

sudo systemctl daemon-reload
sudo systemctl enable crawler.service
sudo systemctl start crawler.service

Check status with:

sudo systemctl status crawler.service

For your ERROR condition check, modify your PHP script to return proper exit codes:

<?php
try {
    // Your crawler logic
    exit(0); // Success
} catch (Exception $e) {
    exit(1); // Error
}

For more control, consider these additions to your service file:

# Log configuration
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=crawler

# Resource limits
LimitNOFILE=65536
MemoryLimit=500M

If your service fails to start:

journalctl -u crawler.service -b

This shows detailed logs from systemd about your service's behavior.

html

When migrating from Ubuntu 14.04 to 16.04, many developers encounter issues with service management due to the transition from Upstart to Systemd. The original Upstart configuration files (*.conf in /etc/init) no longer work, resulting in the "Unit not found" error.

For Ubuntu 16.04 and later, we need to create a systemd service unit file instead. Here's how to properly implement your PHP crawler as a daemon:

[Unit]
Description=PHP Crawler Daemon
After=network.target

[Service]
Type=simple
User=www-data
ExecStart=/usr/bin/php -f /var/www/crawler.php
Restart=always
RestartSec=5
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=php-crawler

[Install]
WantedBy=multi-user.target

1. Create the service file:

   sudo nano /etc/systemd/system/crawler.service

2. Paste the above configuration
3. Enable and start the service:

   sudo systemctl daemon-reload
   sudo systemctl enable crawler.service
   sudo systemctl start crawler.service

For more complex scenarios, consider these additional parameters:

# Environment variables
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# Resource limits
LimitNOFILE=65536
LimitNPROC=2048

# Working directory
WorkingDirectory=/var/www/

Check your daemon's status and logs:

sudo systemctl status crawler.service
journalctl -u crawler.service -f

For PHP scripts that might hang, add a timeout:

TimeoutSec=300

When moving from Upstart to Systemd:

• Remove old Upstart configs from /etc/init/
• Convert any Upstart-specific directives
• Test the new service thoroughly