Implementing Multi-Application Health Checks for AWS ELB When Hosting Multiple Services Per EC2 Instance


8 views

When running multiple applications (like microservices) on a single EC2 instance behind an Elastic Load Balancer, the default health check mechanism presents limitations. AWS ELB only allows configuring one health check endpoint per target group, while we need visibility into each application's status.

Here are three proven approaches to solve this:

1. Health Check Aggregator Endpoint

Create a dedicated health endpoint that programmatically checks all applications:


// Node.js example using Express
const express = require('express');
const axios = require('axios');
const app = express();

const SERVICES = [
  { name: 'auth', url: 'http://localhost:3001/health' },
  { name: 'payment', url: 'http://localhost:3002/health' }
];

app.get('/aggregated-health', async (req, res) => {
  const results = await Promise.all(
    SERVICES.map(async service => {
      try {
        const response = await axios.get(service.url);
        return { [service.name]: response.status === 200 };
      } catch {
        return { [service.name]: false };
      }
    })
  );
  
  const allHealthy = results.every(r => Object.values(r)[0]);
  res.status(allHealthy ? 200 : 503).json({ services: results });
});

2. Custom CloudWatch Metrics with Lambda

Deploy a Lambda function that:

  • Polls all application health endpoints
  • Pushes custom metrics to CloudWatch
  • Triggers Auto Scaling actions based on composite health

3. Service Mesh Integration

For advanced implementations, consider using AWS App Mesh:


# Example App Mesh virtual node health check config
virtualNodes:
  - name: auth-service
    listeners:
      - healthCheck:
          protocol: http
          path: /health
          healthyThreshold: 2
          unhealthyThreshold: 2
          timeoutMillis: 5000
          intervalMillis: 10000
  • Health check endpoints should require minimal dependencies
  • Implement circuit breakers to prevent cascading failures
  • Set appropriate timeouts (shorter than ELB's timeout)
  • Include version information in health responses

Use this CloudFormation snippet to create alarms for multi-service health:


Resources:
  MultiServiceAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Composite health check alarm"
      Metrics:
        - Id: "m1"
          Expression: "IF(m1 AND m2, 1, 0)"
          Label: "CompositeHealth"
        - Id: "m1"
          MetricStat:
            Metric:
              Namespace: "Custom"
              MetricName: "AuthServiceHealth"
            Period: 60
            Stat: "Minimum"
        - Id: "m2"
          MetricStat:
            Metric:
              Namespace: "Custom"
              MetricName: "PaymentServiceHealth"
            Period: 60
            Stat: "Minimum"
      ComparisonOperator: "LessThanThreshold"
      Threshold: 1
      EvaluationPeriods: 2

When running multiple applications on a single EC2 instance behind an Elastic Load Balancer (ELB), you'll quickly encounter a limitation: ELB health checks can only monitor one endpoint per target group. This becomes problematic when different applications have different health requirements or when you need granular visibility into each application's status.

Here are three practical approaches to implement comprehensive health monitoring:

1. Custom Health Check Endpoint

Create a dedicated health check endpoint that aggregates status from all applications:


from flask import Flask, jsonify
import requests

app = Flask(__name__)

@app.route('/health')
def health_check():
    status = {
        'app1': check_app1(),
        'app2': check_app2(),
        'overall': True
    }
    
    # If any app fails, mark overall as unhealthy
    status['overall'] = all(status.values())
    
    return jsonify(status), 200 if status['overall'] else 503

def check_app1():
    try:
        response = requests.get('http://localhost:8001/health', timeout=2)
        return response.status_code == 200
    except:
        return False

def check_app2():
    try:
        response = requests.get('http://localhost:8002/health', timeout=2)
        return response.json().get('status') == 'OK'
    except:
        return False

2. Application Load Balancer with Multiple Target Groups

For ALB (not classic ELB), you can create:

  • Separate target groups for each application
  • Different health check paths for each target group
  • Routing rules based on path or host header

3. Custom Lambda Health Check

Implement a Lambda function that performs comprehensive checks and updates instance health via API:


const AWS = require('aws-sdk');
const elb = new AWS.ELBv2();

exports.handler = async (event) => {
    const instanceId = event.instanceId;
    const checks = {
        app1: await checkApp1(),
        app2: await checkApp2()
    };
    
    const isHealthy = checks.app1 && checks.app2;
    
    await elb.setTargetHealth({
        TargetGroupArn: process.env.TARGET_GROUP_ARN,
        Target: { Id: instanceId },
        HealthStatus: isHealthy ? 'healthy' : 'unhealthy'
    }).promise();
    
    return { status: isHealthy ? 'healthy' : 'unhealthy', checks };
};
  • Timeout Handling: Set conservative timeouts for individual application checks
  • Circuit Breakers: Implement fail-fast mechanisms when downstream services are unavailable
  • Logging: Log detailed health check failures for troubleshooting
  • Security: Protect health endpoints with proper authentication

Complement your solution with CloudWatch alarms that track:


aws cloudwatch put-metric-alarm \
    --alarm-name "App1-Unhealthy-Hosts" \
    --metric-name "UnHealthyHostCount" \
    --namespace "AWS/ApplicationELB" \
    --dimensions Name=TargetGroup,Value=target-group-arn \
    --statistic "Average" \
    --period 60 \
    --evaluation-periods 1 \
    --threshold 0 \
    --comparison-operator "GreaterThanThreshold"