Monitoring AWS ECS Deployment failures

This post discusses how ECS state change events can be used to monitor deployment failures on ECS. To set the context, I work on a project where we use ECS to deploy containerized applications, and our CircleCI pipeline is responsible for building Docker images, pushing them to AWS ECR, and initiating the ECS deployment using the aws ecs update-service command. Our CircleCI job ends after this command, at which point we considered the deployment successful. However, the deployment isn't truly complete until the new containers are up and running, which created a gap in monitoring, as containers could fail to start due to issues like failed migrations, incorrect configurations, or resource allocation problems.

Relying solely on aws ecs update-service execution was misleading, as it didn't account for failures after the deployment was initiated. To address this, we needed to listen to ECS state change events, particularly for failed deployments. These events provide real-time insight into whether containers failed to start, allowing us to handle issues like failed migrations or resource allocation errors and notify team on Slack for further investigation.

Using EventBridge to Monitor ECS State Changes

Amazon EventBridge is a powerful event bus that can help monitor and respond to various AWS service events., including ECS deployment state changes. When you deploy containerized applications with ECS, ECS deployment state changes are automatically sent to EventBridge, specifically these:

SERVICE_DEPLOYMENT_IN_PROGRESS
The service deployment is in progress. This event is sent for both initial deployments and rollback deployments.

SERVICE_DEPLOYMENT_COMPLETED
The service deployment has completed. This event is sent once a service reaches a steady state after a deployment.

SERVICE_DEPLOYMENT_FAILED
The service deployment has failed. This event is sent for services with deployment circuit breaker logic turned on.

To track failed ECS deployments, we can set up an EventBridge rule that listens for SERVICE_DEPLOYMENT_FAILED events. This captures real-time failure information, allowing us to quickly respond to issues such as failed migrations, configuration errors, or resource limitations. Below is an example of the EventBridge rule used to listen for these failure events. When this rule matches an event, it can trigger AWS Lambda or other services to send alerts to your Slack channel, providing real-time visibility into deployment failures.

{
    "source": ["aws.ecs"],
    "detail-type": ["ECS Deployment State Change"],
    "detail": {
      "eventType": ["ERROR"],
      "eventName": ["SERVICE_DEPLOYMENT_FAILED"]
    }  
}

Here's an example of a failed deployment event that would trigger this rule. This event indicates that a task failed to start during the ECS deployment, potentially due to issues like incorrect configurations or missing dependencies.

{
   "version": "0",
   "id": "ddca6449-b258-46c0-8653-e0e3aEXAMPLE",
   "detail-type": "ECS Deployment State Change",
   "source": "aws.ecs",
   "account": "111122223333",
   "time": "2020-05-23T12:31:14Z",
   "region": "us-west-2",
   "resources": [ 
        "arn:aws:ecs:us-west-2:111122223333:service/default/servicetest"
   ],
   "detail": {
        "eventType": "ERROR", 
        "eventName": "SERVICE_DEPLOYMENT_FAILED",
        "deploymentId": "ecs-svc/123",
        "updatedAt": "2020-05-23T11:11:11Z",
        "reason": "ECS deployment circuit breaker: task failed to start."
   }
}

So, now we are able to capture failed deployment events on eventbridge rule. We now need to set target, where we want EventBridge to send any events that match the event pattern of the rule. In our case, we'll use AWS Lambda. We'll use Lambda to send slack alerts on our configured incoming webhooks. To read more on how you can setup slack webhook, read this later.

Setting Up AWS Lambda to Post Deployment Failure Alerts to Slack

This lambda function will parse the event, determine whether the failure occurred in staging or production, generate a direct URL to the affected ECS service, and send the failure details to a specified Slack channel. For detailed code, you can refer to this Lambda function code on GitHub Gist.

Conclusion and further readings

Monitoring ECS deployments is crucial to ensure that your applications are running smoothly. By using Amazon EventBridge to capture ECS state change events and integrating AWS Lambda with Slack, you can receive real-time notifications whenever a deployment fails.

For further readings, check out the following resources to deepen your understanding of ECS deployment events and the deployment circuit breaker:

• Amazon ECS Service Deployment Events
• Amazon ECS Deployment Circuit Breaker
• Setting up slack webhooks

These documents will help you gain a more in-depth understanding of the ECS deployment lifecycle and the circuit breaker feature that helps in rolling back failed deployments automatically.