How to Automatically Delete Old Files in AWS S3 Based on Object Age


1 views

Managing storage in S3 becomes crucial when dealing with temporary files, logs, or backups. AWS provides a native solution through Lifecycle Policies that automatically transition or expire objects based on rules you define.

Here's how to configure automatic deletion through the UI:

  1. Navigate to your S3 bucket
  2. Select the Management tab
  3. Click Create lifecycle rule
  4. Name your rule (e.g., "30-day-expiration")
  5. Under Lifecycle rule actions, select Expire current versions of objects
  6. Set your desired number of days (e.g., 30)
  7. Click Create rule

For infrastructure-as-code approach:

aws s3api put-bucket-lifecycle-configuration \
  --bucket your-bucket-name \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "AutoDeleteAfter30Days",
        "Status": "Enabled",
        "Filter": {},
        "Expiration": {
          "Days": 30
        }
      }
    ]
  }'

For infrastructure automation:

resource "aws_s3_bucket_lifecycle_configuration" "example" {
  bucket = aws_s3_bucket.example.id

  rule {
    id = "auto-delete-old-files"
    
    expiration {
      days = 30
    }
    
    status = "Enabled"
  }
}

You can target specific prefixes or object tags:

{
  "Rules": [
    {
      "ID": "DeleteLogsAfter7Days",
      "Filter": {
        "Prefix": "logs/"
      },
      "Status": "Enabled",
      "Expiration": {
        "Days": 7
      }
    }
  ]
}
  • Lifecycle rules typically take 24-48 hours to take effect
  • Versioned buckets require separate configuration for non-current versions
  • Test rules with small datasets before applying to production
  • Monitor costs as deletion operations are billable

Check your rules with:

aws s3api get-bucket-lifecycle-configuration \
  --bucket your-bucket-name

Managing object lifecycle in Amazon S3 buckets becomes crucial as storage needs grow. Many developers face the common issue of accumulating obsolete files that should be automatically purged after a certain retention period.

Amazon S3 provides native lifecycle management capabilities through bucket policies. This feature allows you to automatically transition objects to different storage classes or delete them based on age criteria.


{
    "Rules": [
        {
            "ID": "DeleteOldFilesRule",
            "Status": "Enabled",
            "Filter": {},
            "Expiration": {
                "Days": 30
            }
        }
    ]
}

Here's how to set up lifecycle rules through the AWS Management Console:

  1. Navigate to your S3 bucket
  2. Select the "Management" tab
  3. Click "Add lifecycle rule"
  4. Configure rule details (name, scope)
  5. Set expiration period in days
  6. Review and save the rule

For infrastructure-as-code approaches, here's how to implement this using AWS SDK for Python (Boto3):


import boto3

s3 = boto3.client('s3')

lifecycle_config = {
    'Rules': [
        {
            'ID': 'AutoDeleteAfter30Days',
            'Filter': {},
            'Status': 'Enabled',
            'Expiration': {
                'Days': 30
            }
        }
    ]
}

response = s3.put_bucket_lifecycle_configuration(
    Bucket='your-bucket-name',
    LifecycleConfiguration=lifecycle_config
)

When implementing lifecycle policies:

  • Changes may take up to 24 hours to take effect
  • Lifecycle rules apply to the entire bucket unless filtered
  • Consider versioned buckets - you may need additional rules
  • Monitor S3 storage metrics after implementation

If files aren't being deleted as expected:


# Check existing lifecycle policies
aws s3api get-bucket-lifecycle-configuration --bucket your-bucket-name

Verify IAM permissions include s3:PutLifecycleConfiguration and s3:GetLifecycleConfiguration actions.