How to Force AWS CloudFormation to Recreate a Manually Deleted EC2 Instance


2 views

When working with AWS CloudFormation, you might encounter a frustrating scenario: you manually terminate an EC2 instance that was created by your stack, and CloudFormation doesn't recreate it during stack updates. This is different from Auto Scaling Groups, which automatically replace terminated instances.

The error message typically looks like this:

UPDATE_FAILED   AWS::EC2::Instance  DnsServer1  i-014eee8720c4fb542 does not exist

CloudFormation maintains a state of your infrastructure. When you manually delete a resource:

  • CloudFormation still believes the resource exists in its state
  • During updates, it tries to modify the (now non-existent) resource
  • The update fails because the physical resource is gone

Option 1: Use CloudFormation Stack Policies

You can temporarily modify the stack policy to allow resource replacement:

aws cloudformation set-stack-policy \
  --stack-name YourStackName \
  --stack-policy-body file://policy.json

Where policy.json contains:

{
  "Statement" : [
    {
      "Effect" : "Allow",
      "Action" : "Update:*",
      "Principal": "*",
      "Resource" : "*"
    }
  ]
}

Option 2: Modify the Resource to Force Replacement

Update your template to change a property that forces replacement. For EC2 instances, common options include:

"DnsServer1": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    "InstanceType": {"Ref": "InstanceType"},
    "UserData": {"Fn::Base64": {"Fn::Join": ["", [
      "#!/bin/bash\n",
      "echo '", {"Ref": "ForceReplacementToken"}, "'\n",
      "# Your existing user data here"
    ]]}},
    # Other properties...
  }
}

Then update your stack with a new ForceReplacementToken parameter value.

Option 3: Use CloudFormation Drift Detection

You can detect the drift and then update the stack:

# Detect drift
aws cloudformation detect-stack-drift --stack-name YourStackName

# Wait for drift detection to complete, then:
aws cloudformation update-stack \
  --stack-name YourStackName \
  --use-previous-template \
  --parameters ParameterKey=ForceReplacementToken,ParameterValue=new-value

To prevent this situation:

  • Always manage resources through CloudFormation
  • For testing, consider using nested stacks for components you frequently modify
  • Implement proper IAM policies to prevent manual deletions

If you frequently need this level of control, consider infrastructure-as-code tools that offer more flexibility:

// AWS CDK example
const instance = new ec2.Instance(this, 'Instance', {
  // ... configuration
});

// Force replacement
instance.instance.instanceType = new ec2.InstanceType('t3.micro');

When working with AWS CloudFormation, one common frustration arises when resources are manually deleted outside the stack management. Unlike Auto Scaling Groups that automatically replace terminated instances, standalone EC2 instances remain deleted until explicitly addressed.

The fundamental problem occurs because CloudFormation maintains state consistency. When you manually terminate an EC2 instance that was created through CloudFormation:

  • CloudFormation's state becomes out of sync with actual AWS resources
  • Subsequent stack updates fail because the template references a non-existent resource
  • The stack enters UPDATE_ROLLBACK_COMPLETE state

Here are the most effective approaches to force recreation of manually deleted resources:

Method 1: CloudFormation Stack Update with Modification

Update the resource definition in your template to force recreation:

Resources:
  DnsServer1:
    Type: AWS::EC2::Instance
    Properties:
      # Add or modify any property to force update
      UserData: 
        Fn::Base64: !Sub |
          #!/bin/bash
          echo "Updated at ${AWS::StackName}"

Method 2: Use CloudFormation Drift Detection

CloudFormation's drift detection can help identify mismatches:

aws cloudformation detect-stack-drift --stack-name your-stack-name

Method 3: Resource Import Workaround

For critical scenarios where you can't modify the template:

  1. Create a new EC2 instance with the same configuration
  2. Import it into your stack using:
aws cloudformation import-resources-to-stack \
    --stack-name your-stack-name \
    --resources '[
      {
        "ResourceType": "AWS::EC2::Instance",
        "LogicalResourceId": "DnsServer1",
        "ResourceIdentifier": {"InstanceId": "i-newinstanceid"}
      }
    ]'

To prevent similar issues:

  • Avoid manual changes to CloudFormation-managed resources
  • Implement IAM policies restricting manual deletion
  • Consider using AWS Service Catalog for production environments
  • For test environments, implement automatic stack recreation pipelines

For critical instances that must auto-recover, implement a custom resource:

Resources:
  DnsServer1Recovery:
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt RecoveryFunction.Arn
      InstanceType: t3.medium
      # Other required properties

With corresponding Lambda function to monitor and recreate instances.