How to Force Recreate AWS EC2 Instances and RDS Databases in CloudFormation Stacks


1 views

When working with AWS CloudFormation, certain resources like AWS::EC2::Instance and AWS::RDS::DBInstance are designed to be modified in-place rather than recreated. This becomes problematic when:

  • RDS engine version downgrades are impossible (e.g., MySQL 5.6 → 5.5)
  • EC2 UserData scripts need fresh execution
  • Underlying resource corruption occurs

While AWS recommends stack deletion/recreation for such cases, production environments often demand better solutions:

# Example of problematic RDS configuration
Resources:
  MyDatabase:
    Type: AWS::RDS::DBInstance
    Properties:
      Engine: mysql
      EngineVersion: "5.6"  # Cannot be changed to 5.5 later
      DBInstanceClass: db.t2.micro

1. The Availability Zone Trick

Changing the AZ forces recreation while maintaining the stack:

Resources:
  MyEC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: us-east-1a  # Change to us-east-1b to trigger recreate
      ...

2. Logical ID Modification

Renaming the resource in template forces new creation (existing resource becomes orphaned):

Resources:
  MyEC2Instance_V2:  # Changed from MyEC2Instance
    Type: AWS::EC2::Instance
    Properties: ...

3. Instance Replacement Properties

For EC2 instances, modify these recreation-triggering properties:

  • ImageId
  • InstanceType
  • SubnetId (when in VPC)

Custom Resource Lambda Function

For controlled recreation without stack updates:

Resources:
  RecreateTrigger:
    Type: Custom::Recreator
    Properties:
      ServiceToken: !GetAtt RecreateLambda.Arn
      ResourceType: "EC2"
      ResourceId: !Ref MyEC2Instance

ASG with UpdatePolicy

For production workloads, consider using Auto Scaling Groups:

Resources:
  MyASG:
    Type: AWS::AutoScaling::AutoScalingGroup
    UpdatePolicy:
      AutoScalingReplacingUpdate:
        WillReplace: true
  • Always snapshot RDS before attempting recreation
  • For stateful instances, implement proper backup solutions
  • Consider blue/green deployment patterns for critical workloads
  • Monitor orphaned resources that may incur costs

When working with AWS CloudFormation, certain resource types like AWS::EC2::Instance and AWS::RDS::DBInstance present a unique challenge - they're designed to be modified in-place rather than recreated during stack updates. This becomes problematic when you need to force a full recreation due to:

  • Engine version downgrades (like MySQL 5.6→5.5)
  • UserData script changes requiring fresh execution
  • Stuck in UPDATE_FAILED states

CloudFormation follows specific update rules based on property changes:

# Sample RDS instance showing immutable properties
Resources:
  MyDB:
    Type: AWS::RDS::DBInstance
    Properties:
      Engine: mysql
      EngineVersion: "5.6"  # Changing this requires replacement
      DBInstanceClass: db.t2.micro

The key indicator is whether a property requires Replacement in the resource documentation.

1. Changing Immutable Properties

# Force EC2 recreation by modifying NetworkInterfaces
Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      SubnetId: !Ref SubnetA  # Change to SubnetB to force replacement
      UserData: 
        Fn::Base64: !Sub |
          #!/bin/bash
          yum update -y
          # Your new script here

2. Using AWS::CloudFormation::Init with Metadata

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /etc/script.sh:
              content: !Sub |
                #!/bin/bash
                echo "New script version"
    Properties:
      UserData:
        Fn::Base64: !Sub |
          #!/bin/bash
          /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance

3. The Availability Zone Workaround

As mentioned in the original post, changing AZs forces recreation:

# For RDS instances
Resources:
  MyDB:
    Type: AWS::RDS::DBInstance
    Properties:
      AvailabilityZone: us-east-1a  # Change to us-east-1b

Using Custom Resources with Lambda

Resources:
  CustomEC2Replacer:
    Type: Custom::EC2Replacer
    Properties:
      ServiceToken: !GetAtt ReplacementLambda.Arn
      InstanceId: !Ref MyInstance
      ForceRecreate: true  # Toggle this to force recreation

  ReplacementLambda:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.8
      Handler: index.handler
      Code:
        ZipFile: |
          import boto3
          def handler(event, context):
              if event['RequestType'] == 'Delete':
                  return {'Status': 'SUCCESS'}
              
              ec2 = boto3.resource('ec2')
              instance = ec2.Instance(event['ResourceProperties']['InstanceId'])
              if event['ResourceProperties'].get('ForceRecreate'):
                  instance.terminate()
                  instance.wait_until_terminated()
              return {'Status': 'SUCCESS'}

The Nuclear Option: Stack Policies

Create a stack policy that prevents updates to certain resources, forcing replacement:

{
  "Statement" : [
    {
      "Effect" : "Deny",
      "Action" : ["Update:Modify"],
      "Principal": "*",
      "Resource" : "LogicalResourceId/MyDB"
    }
  ]
}
  • Version-control all UserData scripts externally
  • Use CloudFormation parameters for critical properties
  • Consider blue/green deployments for production systems
  • Document all forced-recreation triggers in team runbooks

Remember that while these techniques work, they should be used judiciously in production environments. Always test recreation scenarios in staging first.