How to Properly Shutdown an EC2 Instance from Within Ubuntu: Solving the 50% CPU Hang Issue


6 views

Many AWS users running Ubuntu instances encounter this frustrating scenario: you issue a standard shutdown command like sudo poweroff, see the normal shutdown messages, but the instance remains running with 50% CPU usage. Let's dig into why this happens and how to solve it properly.

The issue typically stems from how the hypervisor interacts with the virtualized hardware. When you run:

sudo poweroff
# Or alternatively
sudo shutdown -h now

The system attempts to send ACPI power signals that sometimes don't properly reach the Xen hypervisor (especially on older Ubuntu AMIs). This leaves the instance in a zombie state where processes are terminated but the VM itself isn't properly stopped.

Here are three proven approaches that work consistently across different Ubuntu AMIs:

Method 1: Using the AWS CLI

The most reliable way is to have the instance terminate itself through the AWS API:

# First install AWS CLI if not present
sudo apt-get install awscli
# Then configure with proper credentials
aws configure

# Self-termination command
aws ec2 stop-instances --instance-ids $(curl -s http://169.254.169.254/latest/meta-data/instance-id) --region us-west-2

Method 2: Forcing a Hard Shutdown

If you prefer a local solution without API calls, this sequence often works:

# First attempt graceful shutdown
sudo shutdown -h now

# If that doesn't work after 2 minutes, force it
echo 1 | sudo tee /proc/sys/kernel/sysrq
echo b | sudo tee /proc/sysrq-trigger

Method 3: Cloud-Init Solution

For automated environments, add this to your cloud-init configuration:

power_state:
  mode: poweroff
  timeout: 120
  condition: True

When shutdowns fail, check these logs for clues:

# View system logs
cat /var/log/syslog | grep -i shutdown

# Check cloud-init logs
cat /var/log/cloud-init.log

# Examine kernel messages
dmesg | tail -50

The CPU spike typically occurs when the system is stuck between running and shutdown states. To prevent this:

  1. Ensure all services properly implement Systemd's shutdown handling
  2. Add proper sync operations before shutdown
  3. Consider replacing problematic init systems

Here's a sample Systemd unit that handles shutdown cleanly:

[Unit]
Description=Clean shutdown handler
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target

[Service]
Type=oneshot
ExecStart=/usr/bin/sync
ExecStop=/bin/true
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
  • For production systems, always use the AWS API method
  • For development, the forced shutdown method works well
  • Update to newer Ubuntu AMIs where this issue is resolved

Remember that instance-store backed instances can't be stopped, only terminated. Adjust your shutdown strategy accordingly.

When attempting to shutdown an Ubuntu 10.10 EC2 instance from within the operating system, many developers encounter a frustrating scenario where the instance gets stuck with 50% CPU usage instead of properly terminating. This behavior occurs with commands like:

sudo poweroff
sudo shutdown now
sudo halt -p --verbose

The issue stems from how the Xen hypervisor interacts with certain Ubuntu AMIs. While most AMIs handle shutdown commands gracefully, some get stuck during the poweroff sequence. Key symptoms include:

  • First shutdown command returns to shell without effect
  • Second attempt terminates SSH but leaves instance running
  • Constant 50% CPU usage visible in AWS console
  • System processes killed but instance remains active

After extensive testing, these methods consistently work for problematic AMIs:

# Method 1: Using ec2-stop-instances via AWS CLI
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Method 2: Forced sync and poweroff
sudo sync && sudo echo 1 > /proc/sys/kernel/sysrq && sudo echo o > /proc/sysrq-trigger

# Method 3: Direct halt with forced poweroff
sudo halt --force --force

For instances with IAM permissions, you can trigger self-termination:

# Using the instance metadata service
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | awk '{print $3}' | sed 's/"//g' | sed 's/,//g')

aws ec2 terminate-instances --instance-ids $INSTANCE_ID --region $REGION

To avoid this issue in future deployments:

  1. Update to newer Ubuntu AMIs (16.04 LTS or later)
  2. Create custom AMIs with proper shutdown hooks
  3. Implement CloudWatch Events for instance monitoring
  4. Use Systems Manager for automated instance management

When shutdowns fail, check these logs for clues:

# View kernel messages
dmesg | tail -50

# Check cloud-init logs
cat /var/log/cloud-init.log

# Examine systemd journal
journalctl -b -0 -n 100