AWS ECS Service Failing to Launch Tasks: Debugging EC2 Container Issues


3 views

When your ECS service shows "3 desired tasks" but zero actual tasks running, there's usually one of these fundamental issues at play:

  • Resource allocation conflicts
  • Port mapping collisions
  • IAM permission gaps
  • EC2 instance capacity problems

Your task definition shows a potential red flag with duplicate host ports:

{
  "portMappings": [
    {
      "hostPort": 5000,  // TCP
      "containerPort": 25565
    },
    {
      "hostPort": 5000,  // UDP
      "containerPort": 25565
    }
  ]
}

While technically allowed (different protocols), this often causes deployment issues in practice. Consider either:

  1. Using different host ports
  2. Removing the UDP mapping if not strictly necessary

SSH into your instances and run these diagnostic commands:

# Check ECS agent status
sudo systemctl status ecs

# View recent ECS agent logs
sudo cat /var/log/ecs/ecs-agent.log.[0-9] | grep -i error

# Verify available resources
docker info | grep -iE 'memory|cpu'

Ensure your EC2 instance role has these critical permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:CreateCluster",
        "ecs:DeregisterContainerInstance",
        "ecs:DiscoverPollEndpoint",
        "ecs:Poll",
        "ecs:RegisterContainerInstance",
        "ecs:StartTelemetrySession",
        "ecs:Submit*",
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

Enable ECS Exec for live container inspection:

aws ecs update-service \
  --cluster your-cluster-name \
  --service your-service-name \
  --enable-execute-command \
  --region us-west-1

Then connect to any running task:

aws ecs execute-command \
  --cluster your-cluster-name \
  --task task-id \
  --container BungeeCord \
  --interactive \
  --command "/bin/sh"

When your ECS service shows "3 desired tasks" but zero actually running, there's usually one of five common culprits:

// Typical failure points to check:
1. EC2 instance capacity constraints
2. IAM role permission gaps
3. Port mapping conflicts 
4. Task resource oversubscription
5. Launch type mismatches

Your task definition requests 1024MB memory reservation - verify your EC2 instances actually have available capacity:

# SSH into an instance and check:
$ docker info | grep -i memory
$ free -m
$ cat /proc/meminfo | grep MemAvailable

The port binding conflict jumps out immediately - you're trying to bind both TCP and UDP to the same host port (5000). This creates a race condition during container startup.

Instead of guessing, run these AWS CLI commands to get concrete answers:

# Check service events for deployment errors
aws ecs describe-services --cluster your-cluster --services your-service

# Verify container instance status
aws ecs list-container-instances --cluster your-cluster --status ACTIVE

# Inspect stopped tasks (even if UI shows empty)
aws ecs list-tasks --cluster your-cluster --desired-status STOPPED

Modify your task definition to avoid port collisions:

{
  "containerDefinitions": [
    {
      "portMappings": [
        {
          "hostPort": 5000,  // TCP gets 5000
          "protocol": "tcp",
          "containerPort": 25565
        },
        {
          "hostPort": 5001,  // UDP gets 5001
          "protocol": "udp", 
          "containerPort": 25565
        }
      ]
    }
  ]
}

When instances show as active but won't run tasks:

  • Verify the ECS agent is running: sudo systemctl status ecs
  • Check instance registration: curl http://localhost:51678/v1/metadata
  • Confirm agent can pull images: docker pull itzg/bungeecord

Sometimes you need to completely reset the deployment:

# Force new deployment
aws ecs update-service --cluster your-cluster \
  --service your-service \
  --force-new-deployment

# Alternatively, delete and recreate
aws ecs delete-service --cluster your-cluster --service your-service
aws ecs create-service --cluster your-cluster --service-name your-service \
  --task-definition BungeeCordTask:3 --desired-count 3