Many AWS users don't realize that stopped EC2 instances continue to incur costs through attached EBS volumes. At $0.10/GB-month (gp3), a single 1TB volume costs $100 monthly even when the instance isn't running. For environments with hundreds of instances, these forgotten resources can silently accumulate significant costs.
Here are three technical approaches to determine when instances were last stopped:
1. AWS CLI with Instance State Transition
The most accurate method uses CloudTrail logs to find the StopInstances API call timestamp:
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=StopInstances \
--query "Events[?contains(Resources[].ResourceName, 'i-1234567890abcdef0')].{Time:EventTime}" \
--output text
2. AWS Systems Manager (SSM) Inventory
For instances with SSM agent installed, check the last ping time:
aws ssm describe-instance-information \
--filters "Key=InstanceIds,Values=i-1234567890abcdef0" \
--query "InstanceInformationList[].{LastPing:LastPingDateTime}" \
--output text
3. EC2 Instance State Transition Metrics
CloudWatch metrics provide state transition history (14-month retention):
aws cloudwatch get-metric-data \
--metric-data-queries '[
{
"Id": "m1",
"MetricStat": {
"Metric": {
"Namespace": "AWS/EC2",
"MetricName": "StatusCheckFailed_Instance",
"Dimensions": [{"Name": "InstanceId", "Value": "i-1234567890abcdef0"}]
},
"Period": 86400,
"Stat": "Maximum"
},
"ReturnData": true
}
]' \
--start-time $(date -d "90 days ago" +%Y-%m-%dT%H:%M:%S) \
--end-time $(date +%Y-%m-%dT%H:%M:%S)
This Python script identifies instances stopped for over 30 days and creates termination reports:
import boto3
from datetime import datetime, timedelta
ec2 = boto3.client('ec2')
cloudtrail = boto3.client('cloudtrail')
def get_stopped_instances():
response = ec2.describe_instances(
Filters=[{'Name': 'instance-state-name', 'Values': ['stopped']}]
)
return [i['InstanceId'] for r in response['Reservations'] for i in r['Instances']]
def get_last_stop_time(instance_id):
events = cloudtrail.lookup_events(
LookupAttributes=[{'AttributeKey': 'ResourceName', 'AttributeValue': instance_id}],
MaxResults=50
)
for event in events['Events']:
if event['EventName'] == 'StopInstances':
return event['EventTime']
return None
def main():
threshold = datetime.now() - timedelta(days=30)
candidates = []
for instance_id in get_stopped_instances():
stop_time = get_last_stop_time(instance_id)
if stop_time and stop_time.replace(tzinfo=None) < threshold:
candidates.append((instance_id, stop_time))
print("Instances stopped before", threshold)
for instance_id, stop_time in sorted(candidates, key=lambda x: x[1]):
print(f"{instance_id}: {stop_time}")
if __name__ == "__main__":
main()
Before terminating instances:
- Check CloudWatch alarms that might reference the instance
- Verify no Auto Scaling groups are associated
- Confirm no EBS snapshots are actively being created
- Check AWS Backup or other protection mechanisms
- Review AWS Cost Explorer to understand actual savings impact
Instead of immediate termination:
# Create EBS snapshots and delete volumes
aws ec2 create-snapshot --volume-id vol-1234567890abcdef0
aws ec2 delete-volume --volume-id vol-1234567890abcdef0
# Or modify volume type to cold storage (sc1)
aws ec2 modify-volume --volume-id vol-1234567890abcdef0 --volume-type sc1
Many AWS users don't realize that stopped EC2 instances continue to incur EBS storage costs ($140/month per TB). Over time, hundreds of these "zombie instances" can accumulate, significantly inflating your AWS bill. The challenge is determining which instances are truly abandoned versus those temporarily stopped for maintenance or migration purposes.
The most reliable method involves querying AWS CloudTrail logs. These contain detailed API call histories, including StopInstances events. Here's a Python script using boto3 to retrieve shutdown timestamps:
import boto3
from datetime import datetime, timedelta
def get_shutdown_events(instance_id, days_to_check=90):
cloudtrail = boto3.client('cloudtrail')
# Set time range
end_time = datetime.utcnow()
start_time = end_time - timedelta(days=days_to_check)
response = cloudtrail.lookup_events(
LookupAttributes=[
{
'AttributeKey': 'ResourceName',
'AttributeValue': instance_id
},
{
'AttributeKey': 'EventName',
'AttributeValue': 'StopInstances'
}
],
StartTime=start_time,
EndTime=end_time,
MaxResults=50
)
shutdown_times = []
for event in response['Events']:
shutdown_times.append(event['EventTime'])
return sorted(shutdown_times, reverse=True)
# Example usage
instance_ids = ['i-1234567890abcdef0', 'i-11111111111111111']
for instance_id in instance_ids:
shutdowns = get_shutdown_events(instance_id)
print(f"Instance {instance_id} was last stopped at: {shutdowns[0] if shutdowns else 'Never (or before lookback period)'}")
If CloudTrail isn't enabled in your account, consider these alternatives:
1. AWS Config Rules: If AWS Config is set up, you can query configuration history:
aws configservice get-resource-config-history \
--resource-type AWS::EC2::Instance \
--resource-id i-1234567890abcdef0 \
--chronological-order Reverse \
--query 'configurationItems[?configuration.state.name==stopped]'
2. EC2 Instance Metadata: For currently running instances, check the system log for unexpected reboots that might indicate maintenance windows:
ssh ec2-user@your-instance "sudo cat /var/log/messages | grep -i 'shutdown'"
Combine these techniques to create an automated cleanup system:
def evaluate_instance_for_termination(instance_id):
# Get shutdown history
shutdowns = get_shutdown_events(instance_id)
# Check if stopped longer than threshold (e.g., 60 days)
if not shutdowns:
return False # Instance may be running or never stopped
last_shutdown = shutdowns[0]
threshold = datetime.utcnow() - timedelta(days=60)
if last_shutdown < threshold:
# Additional safety checks
ec2 = boto3.resource('ec2')
instance = ec2.Instance(instance_id)
# Skip instances with special tags
if 'DoNotTerminate' in [t['Key'] for t in instance.tags]:
return False
# Check for recent snapshots
snapshots = list(ec2.snapshots.filter(
Filters=[{'Name': 'volume-id', 'Values': [vol.id for vol in instance.volumes.all()]}]
))
if not snapshots: # No recent backups found
return True
return False
- Always create final snapshots before termination
- Implement a notification system for affected teams
- Consider implementing a staging process (e.g., tag first, terminate later)
- Monitor AWS Savings Plan/RI coverage before major cleanup operations