When working with AWS CloudFormation, you might encounter a frustrating scenario: you manually terminate an EC2 instance that was created by your stack, and CloudFormation doesn't recreate it during stack updates. This is different from Auto Scaling Groups, which automatically replace terminated instances.
The error message typically looks like this:
UPDATE_FAILED AWS::EC2::Instance DnsServer1 i-014eee8720c4fb542 does not exist
CloudFormation maintains a state of your infrastructure. When you manually delete a resource:
- CloudFormation still believes the resource exists in its state
- During updates, it tries to modify the (now non-existent) resource
- The update fails because the physical resource is gone
Option 1: Use CloudFormation Stack Policies
You can temporarily modify the stack policy to allow resource replacement:
aws cloudformation set-stack-policy \
--stack-name YourStackName \
--stack-policy-body file://policy.json
Where policy.json contains:
{
"Statement" : [
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
}
]
}
Option 2: Modify the Resource to Force Replacement
Update your template to change a property that forces replacement. For EC2 instances, common options include:
"DnsServer1": {
"Type": "AWS::EC2::Instance",
"Properties": {
"InstanceType": {"Ref": "InstanceType"},
"UserData": {"Fn::Base64": {"Fn::Join": ["", [
"#!/bin/bash\n",
"echo '", {"Ref": "ForceReplacementToken"}, "'\n",
"# Your existing user data here"
]]}},
# Other properties...
}
}
Then update your stack with a new ForceReplacementToken parameter value.
Option 3: Use CloudFormation Drift Detection
You can detect the drift and then update the stack:
# Detect drift
aws cloudformation detect-stack-drift --stack-name YourStackName
# Wait for drift detection to complete, then:
aws cloudformation update-stack \
--stack-name YourStackName \
--use-previous-template \
--parameters ParameterKey=ForceReplacementToken,ParameterValue=new-value
To prevent this situation:
- Always manage resources through CloudFormation
- For testing, consider using nested stacks for components you frequently modify
- Implement proper IAM policies to prevent manual deletions
If you frequently need this level of control, consider infrastructure-as-code tools that offer more flexibility:
// AWS CDK example
const instance = new ec2.Instance(this, 'Instance', {
// ... configuration
});
// Force replacement
instance.instance.instanceType = new ec2.InstanceType('t3.micro');
When working with AWS CloudFormation, one common frustration arises when resources are manually deleted outside the stack management. Unlike Auto Scaling Groups that automatically replace terminated instances, standalone EC2 instances remain deleted until explicitly addressed.
The fundamental problem occurs because CloudFormation maintains state consistency. When you manually terminate an EC2 instance that was created through CloudFormation:
- CloudFormation's state becomes out of sync with actual AWS resources
- Subsequent stack updates fail because the template references a non-existent resource
- The stack enters UPDATE_ROLLBACK_COMPLETE state
Here are the most effective approaches to force recreation of manually deleted resources:
Method 1: CloudFormation Stack Update with Modification
Update the resource definition in your template to force recreation:
Resources:
DnsServer1:
Type: AWS::EC2::Instance
Properties:
# Add or modify any property to force update
UserData:
Fn::Base64: !Sub |
#!/bin/bash
echo "Updated at ${AWS::StackName}"
Method 2: Use CloudFormation Drift Detection
CloudFormation's drift detection can help identify mismatches:
aws cloudformation detect-stack-drift --stack-name your-stack-name
Method 3: Resource Import Workaround
For critical scenarios where you can't modify the template:
- Create a new EC2 instance with the same configuration
- Import it into your stack using:
aws cloudformation import-resources-to-stack \
--stack-name your-stack-name \
--resources '[
{
"ResourceType": "AWS::EC2::Instance",
"LogicalResourceId": "DnsServer1",
"ResourceIdentifier": {"InstanceId": "i-newinstanceid"}
}
]'
To prevent similar issues:
- Avoid manual changes to CloudFormation-managed resources
- Implement IAM policies restricting manual deletion
- Consider using AWS Service Catalog for production environments
- For test environments, implement automatic stack recreation pipelines
For critical instances that must auto-recover, implement a custom resource:
Resources:
DnsServer1Recovery:
Type: AWS::CloudFormation::CustomResource
Properties:
ServiceToken: !GetAtt RecoveryFunction.Arn
InstanceType: t3.medium
# Other required properties
With corresponding Lambda function to monitor and recreate instances.