When working with AWS CloudFormation, certain resources like AWS::EC2::Instance
and AWS::RDS::DBInstance
are designed to be modified in-place rather than recreated. This becomes problematic when:
- RDS engine version downgrades are impossible (e.g., MySQL 5.6 → 5.5)
- EC2 UserData scripts need fresh execution
- Underlying resource corruption occurs
While AWS recommends stack deletion/recreation for such cases, production environments often demand better solutions:
# Example of problematic RDS configuration
Resources:
MyDatabase:
Type: AWS::RDS::DBInstance
Properties:
Engine: mysql
EngineVersion: "5.6" # Cannot be changed to 5.5 later
DBInstanceClass: db.t2.micro
1. The Availability Zone Trick
Changing the AZ forces recreation while maintaining the stack:
Resources:
MyEC2Instance:
Type: AWS::EC2::Instance
Properties:
AvailabilityZone: us-east-1a # Change to us-east-1b to trigger recreate
...
2. Logical ID Modification
Renaming the resource in template forces new creation (existing resource becomes orphaned):
Resources:
MyEC2Instance_V2: # Changed from MyEC2Instance
Type: AWS::EC2::Instance
Properties: ...
3. Instance Replacement Properties
For EC2 instances, modify these recreation-triggering properties:
ImageId
InstanceType
SubnetId
(when in VPC)
Custom Resource Lambda Function
For controlled recreation without stack updates:
Resources:
RecreateTrigger:
Type: Custom::Recreator
Properties:
ServiceToken: !GetAtt RecreateLambda.Arn
ResourceType: "EC2"
ResourceId: !Ref MyEC2Instance
ASG with UpdatePolicy
For production workloads, consider using Auto Scaling Groups:
Resources:
MyASG:
Type: AWS::AutoScaling::AutoScalingGroup
UpdatePolicy:
AutoScalingReplacingUpdate:
WillReplace: true
- Always snapshot RDS before attempting recreation
- For stateful instances, implement proper backup solutions
- Consider blue/green deployment patterns for critical workloads
- Monitor orphaned resources that may incur costs
When working with AWS CloudFormation, certain resource types like AWS::EC2::Instance
and AWS::RDS::DBInstance
present a unique challenge - they're designed to be modified in-place rather than recreated during stack updates. This becomes problematic when you need to force a full recreation due to:
- Engine version downgrades (like MySQL 5.6→5.5)
- UserData script changes requiring fresh execution
- Stuck in UPDATE_FAILED states
CloudFormation follows specific update rules based on property changes:
# Sample RDS instance showing immutable properties
Resources:
MyDB:
Type: AWS::RDS::DBInstance
Properties:
Engine: mysql
EngineVersion: "5.6" # Changing this requires replacement
DBInstanceClass: db.t2.micro
The key indicator is whether a property requires Replacement in the resource documentation.
1. Changing Immutable Properties
# Force EC2 recreation by modifying NetworkInterfaces
Resources:
MyInstance:
Type: AWS::EC2::Instance
Properties:
SubnetId: !Ref SubnetA # Change to SubnetB to force replacement
UserData:
Fn::Base64: !Sub |
#!/bin/bash
yum update -y
# Your new script here
2. Using AWS::CloudFormation::Init with Metadata
Resources:
MyInstance:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Init:
config:
files:
/etc/script.sh:
content: !Sub |
#!/bin/bash
echo "New script version"
Properties:
UserData:
Fn::Base64: !Sub |
#!/bin/bash
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource MyInstance
3. The Availability Zone Workaround
As mentioned in the original post, changing AZs forces recreation:
# For RDS instances
Resources:
MyDB:
Type: AWS::RDS::DBInstance
Properties:
AvailabilityZone: us-east-1a # Change to us-east-1b
Using Custom Resources with Lambda
Resources:
CustomEC2Replacer:
Type: Custom::EC2Replacer
Properties:
ServiceToken: !GetAtt ReplacementLambda.Arn
InstanceId: !Ref MyInstance
ForceRecreate: true # Toggle this to force recreation
ReplacementLambda:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.8
Handler: index.handler
Code:
ZipFile: |
import boto3
def handler(event, context):
if event['RequestType'] == 'Delete':
return {'Status': 'SUCCESS'}
ec2 = boto3.resource('ec2')
instance = ec2.Instance(event['ResourceProperties']['InstanceId'])
if event['ResourceProperties'].get('ForceRecreate'):
instance.terminate()
instance.wait_until_terminated()
return {'Status': 'SUCCESS'}
The Nuclear Option: Stack Policies
Create a stack policy that prevents updates to certain resources, forcing replacement:
{
"Statement" : [
{
"Effect" : "Deny",
"Action" : ["Update:Modify"],
"Principal": "*",
"Resource" : "LogicalResourceId/MyDB"
}
]
}
- Version-control all UserData scripts externally
- Use CloudFormation parameters for critical properties
- Consider blue/green deployments for production systems
- Document all forced-recreation triggers in team runbooks
Remember that while these techniques work, they should be used judiciously in production environments. Always test recreation scenarios in staging first.