How to Keep Forever Processes Running After Jenkins Build Completion


2 views

Many developers encounter process termination issues when running long-lived services through Jenkins. The specific case involves using forever to maintain Node.js applications, where the process dies immediately after the build completes despite successful startup during execution.

Jenkins kills all child processes when a build completes by default. This behavior occurs because:

  • Jenkins maintains process trees for each build
  • The shell executor terminates when job finishes
  • All descendant processes receive SIGHUP

Option 1: Using nohup

nohup forever start -a -l /var/log/forever.log app.js > /dev/null 2>&1 &

Option 2: Disconnecting with setsid

setsid forever start -a -l /var/log/forever.log app.js

Option 3: Jenkins-specific Approach

Add this to your Jenkinsfile:

pipeline {
  agent any
  stages {
    stage('Deploy') {
      steps {
        sh '''
          npm install
          forever stop app.js || true
          BUILD_ID=dontKillMe forever start -a -l /var/log/forever.log app.js
        '''
      }
    }
  }
}

For more control, consider these additional measures:

Using Jenkins Process Tree Killer Plugin

// In Jenkins configuration
env.JENKINS_NODE_COOKIE=dontKillMe
sh 'forever start -a -l /var/log/forever.log app.js'

PM2 Alternative

If you're open to alternatives, PM2 handles daemonization better:

npm install -g pm2
pm2 start app.js --name "myapp" --log /var/log/pm2.log
pm2 save
pm2 startup
  • Check /var/log/forever.log for errors
  • Verify Jenkins user permissions on the target files
  • Test commands manually as Jenkins user first
  • Monitor process lists with ps aux | grep node

For production environments:

  • Use init systems (systemd) for critical services
  • Implement proper logging rotation
  • Set up monitoring for the forever processes
  • Consider containerization for better process isolation

When executing Node.js applications via Forever in Jenkins build steps, many developers encounter a frustrating scenario: the process starts successfully during the build but terminates immediately after the job completes. This behavior occurs because Jenkins automatically terminates all child processes when the build finishes.

Jenkins creates a process group for each build. When the build completes, all processes in this group receive termination signals. This includes:

  • Direct shell commands
  • Child processes spawned by those commands
  • Daemon processes started during the build

1. Using nohup with Forever

The most straightforward solution is to use nohup to detach the process:

nohup forever start -a -l /var/log/forever.log app.js &

2. Jenkins Process Tree Killer Configuration

Add these Java options to your Jenkins JVM arguments:

-Dhudson.util.ProcessTree.disable=true

Or configure it per job using the BUILD_ID trick:

BUILD_ID=dontKillMe nohup forever start -a -l /var/log/forever.log app.js &

3. Systemd Service Approach

For more robust solutions, consider creating a systemd service:

[Unit]
Description=Node.js Application
After=network.target

[Service]
ExecStart=/usr/bin/forever start -a -l /var/log/forever.log /path/to/app.js
Restart=always
User=jenkins
Group=jenkins
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target
  • Always log forever output to a file for debugging
  • Consider using process managers like PM2 for better monitoring
  • Set proper file permissions for the Jenkins user
  • Monitor disk space for log files

If processes still terminate unexpectedly:

# Check process owner
ps aux | grep app.js

# Verify file permissions
ls -la /var/log/forever.log

# Examine system logs
journalctl -xe