Mastering AWS Incident Management: Automating Responses with Systems Manager Incident Manager

Overview

When handling increased error rates in AWS Lambda, categorizing errors and defining escalation paths is crucial. This guide demonstrates how to use AWS Systems Manager Incident Manager to automatically handle and escalate incidents effectively. The workflow involves collecting error details using Runbooks and notifying stakeholders through Amazon SNS.

Why Use AWS Systems Manager Incident Manager?

AWS Systems Manager Incident Manager provides centralized management for incident response within AWS environments. Key benefits include:

1. Native AWS Integration: Seamlessly integrates with services like Amazon CloudWatch, AWS Lambda, and Amazon EventBridge.

2. Runbook Automation: Facilitates automated or semi-automated workflows to troubleshoot and address incidents.

3. Multi-Channel Notifications: Supports notifications via Amazon SNS, Slack, and Amazon Chime.

4. Cost Efficiency: A viable alternative to commercial solutions for small-to-medium environments.

Limitations

For large-scale organizations requiring detailed reporting, complex team hierarchies, and multi-layer escalation flows, specialized tools like PagerDuty or ServiceNow may be more appropriate.

Architecture Overview

The architecture monitors AWS Lambda functions for errors using CloudWatch Alarms. Incident Manager automatically creates incidents and executes Runbooks for error handling and notifications.

Error Scenarios

• Error A: Standard incident with email notifications.

• Error B: Critical incident requiring SMS notifications and escalations.

CloudWatch Alarms are configured to distinguish between these error types, triggering specific incident responses accordingly.

---

Step-by-Step Configuration

Step 1: Create CloudWatch Alarms for Lambda Errors

Example Lambda Function:

import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    error_type = event.get("errorType")

    try:
        if error_type == "A":
            logger.error("Error A: A standard exception occurred.")
            raise Exception("Error A occurred")
        elif error_type == "B":
            logger.error("Error B: A critical runtime error occurred.")
            raise RuntimeError("Critical Error B occurred")
        else:
            logger.info("No error triggered.")
            return {"statusCode": 200, "body": "Success"}
    except Exception as e:
        logger.exception("An error occurred: %s", e)
        raise

Configure CloudWatch Metrics and Alarms:

1. Metrics Filters: Create filters for Error A and Error B.

1. Alarms: Link these filters to alarms with appropriate thresholds and periods.

1. Alarm Actions: Set up triggers to initiate Incident Manager workflows.

Step 2: Set Up Incident Manager

1. Enable Incident Manager:

- Navigate to the Incident Manager settings in the AWS Management Console and onboard your account.

Step 3: Configure Notification Contacts

• Email: Notify administrators for Error A.

• SMS: Notify stakeholders for Error B escalation.

Step 4: Define Escalation Plans

• Error A: Email notification followed by SMS if unresolved.

• Error B: Immediate SMS notification.

Step 5: Create a Runbook

Runbook Template:

schemaVersion: '0.3'
description: "Runbook for Lambda error handling."
parameters:
  AlarmName:
    type: String
    description: "Name of the CloudWatch Alarm"
  snsTopicArn:
    type: String
    description: "SNS Topic ARN for notifications"
mainSteps:
  - name: GatherErrorLogs
    action: aws:executeScript
    inputs:
      Runtime: python3.8
      Script: |
        def script_handler(events, context):
            return {"status": "Logs gathered"}
  - name: SendNotification
    action: aws:executeAwsApi
    inputs:
      Service: sns
      Api: Publish
      TopicArn: "{{ snsTopicArn }}"
      Message: "Lambda Errors detected. AlarmName: {{ AlarmName }}"

Step 6: Create Response Plans

• Define separate response plans for Error A and Error B.

• Link Runbooks and notification channels to each response plan.

Step 7: Link CloudWatch Alarms to Incident Manager

• Edit alarm actions to trigger the corresponding Incident Manager response plans.

Demo

Commercial Tools Comparison

Feature	AWS Incident Manager	PagerDuty	ServiceNow
Cost Efficiency	High	Medium	Low
AWS Integration	Seamless	Limited	Limited
Escalation Flexibility	Moderate	High	High
Reporting and Analytics	Basic	Advanced	Advanced

Ideal Use Cases for AWS Incident Manager:

• Small-to-medium environments with AWS-centric architectures.

• Simple escalation and notification needs.

• Cost-sensitive deployments.

---

Conclusion

AWS Systems Manager Incident Manager is a cost-effective tool for incident response in AWS-centric environments. While it lacks some advanced features of commercial solutions, it offers robust integration with AWS services and sufficient functionality for many use cases. Its ease of setup and low cost make it an attractive choice for small to medium-scale operations.

---

References

• AWS Systems Manager Incident Manager

• AWS Lambda Monitoring

• Amazon CloudWatch Alarms

• PagerDuty

• ServiceNow

Comments

[Michelle Simon]

Hi! I recently came across labs.itassist.com/ and found it pretty helpful for learning AWS. They offer practical courses, community discussions, and resources like tutorials and guides.

If you're exploring AWS or looking for new ways to dive deeper, it’s worth checking out!