AWS Serverless: Invoke Lambda function asynchronously and use DLQ for failed events

In my previous article, I explained how to set up Lambda destinations for handling failed events, demonstrating how to store these events in an S3 bucket. AWS Lambda destinations also support other options such as SQS, SNS, EventBridge, or another Lambda function.

In this article, I will demonstrate an alternative approach to managing failed events: using a Dead Letter Queue (DLQ) with Amazon SQS. With a DLQ, failed events are captured in a SQS queue, providing a mechanism for troubleshooting and recovery.

Both approaches—using a DLQ or a destination—require asynchronous invocation of the Lambda function. For this demonstration, I will use the AWS CLI to invoke the function asynchronously.

You will notice similarities with the previous article, as the key difference lies in configuring a DLQ instead of a destination to store failed events.

If you missed the earlier article on setting up destinations for failed events, you can watch it using the link provided in the description below:

AWS Serverless: Invoke Lambda function asynchronously and use destination for failed events

Let's look at the architecture diagram!

Review Lambda Function - Hello from Lambda!

The primary purpose of this video is to demonstrate how DLQ work for AWS Lambda functions, with a specific focus on configuring a Dead Letter Queue for failed events. To keep it simple, the Lambda function will implement a basic "Hello, World" example that returns a "Hello from Lambda!" message. This baseline function will serve as the foundation for demonstrating the DLQ feature.

For this walkthrough, I will use the AWS Management Console to create the function and then add the DLQ configuration.

• Navigate to the AWS Management Console for Lambda functions and click Create Function.

• Create a function named HelloWorldGB as shown below.

For this function, I am using Python Runtime 3.13.

Example code is shown in the screenshot below.

This view is from the updated Lambda code editor. If you’re unfamiliar with this console, check out my video that explains how to build a Lambda function using the new editor and SAM templates. The link to the video is in the description.

Add Dead Letter Queue – SQS

Let's first create a SQS queue. This queue will be used as DLQ.

• To create the queue, Navigate to the SQS console and create a new queue named HelloWorldDLQ.
• Make sure that the Lambda function has the necessary permissions to for the SQS queue. Without this permission, you will encounter an error when the function tries to write events to the DLQ.

From the Lambda console, configure dead letter queue for the HelloWorldGB lambda function.

This DLQ will store the failed events. To configure the dead letter queue, navigate to the configuration tab on your lambda function. From there, click on asynchronous invocation.

You can then edit the configuration and add the HelloWorldDLQ.

Add code to raise exception in Lambda

Since the DLQ is configured for failure scenarios, it will only log events when the Lambda function encounters a failure. To test this, I will update the code to intentionally raise an exception, ensuring the function fails and generates a qualifying event.

Invoke the Lambda function after making this change. Due to the forced exception, you should see an error message, as shown in the screenshot below.

Review CloudWatch Log

Next, review the CloudWatch logs. Since the Lambda function has been invoked, a CloudWatch log group will be created. The logs will contain details about the invocation, including the error.

Review Dead Letter Queue - SQS

Navigate to the DLQ t that was configured for this lambda function and check whether the failed event has been posted there.

As you can see, while the lambda function got invoked and generated a failure, the failed event is not logged in the targeted DLQ.

Reason for this is that DLQ function only works when Lambda function is invoked asynchronously. It will not work when function is being invoked synchronously.

Invoke Lambda using AWS CLI

A Lambda function can be invoked asynchronously through various AWS service integrations, such as EventBridge or S3. In the example command below, I demonstrate how to invoke it asynchronously using an AWS CLI command:

` aws lambda invoke \
--function-name arn:aws:lambda:us-east-1:2xxxxxxx7:function:HelloWorldGB \
--invocation-type Event \
--cli-binary-format raw-in-base64-out \
--payload '{ "name": "GB" }' \
--region us-east-1 \
response.Json

Result/response

{
    "StatusCode": 202
}

`
Review the Dead Letter Queue - SQS Again

Now that the result shows a status code of 202, it indicates that the asynchronous call to Lambda was successful. I will now return to the DLQ - SQS to review the results.

As shown in the DLQ screenshot below, the SQS now contains the logged failed event. This event can be further analyzed to identify and understand the cause of the failure.

Cleanup - Delete the Lambda function & DLQ

Once you have completed the setup, ensure you delete the Lambda function to avoid unnecessary resource usage. Additionally, delete the SQS queue and CloudWatch log group. If you created any new roles during the process, remember to delete those as well.

Conclusion

In this article, I demonstrated how to configure a Dead Letter Queue using SQS for a Lambda function.

Additionally, I showcased how to invoke the lambda function asynchronously using AWS CLI command.

I hope you found this article helpful and informative!

Thank you for reading!

Watch the video here:

https://www.youtube.com/watch?v=ifuQc6Bo3B8

𝒢𝒾𝓇𝒾𝓈𝒽 ℬ𝒽𝒶𝓉𝒾𝒶
𝘈𝘞𝘚 𝘊𝘦𝘳𝘵𝘪𝘧𝘪𝘦𝘥 𝘚𝘰𝘭𝘶𝘵𝘪𝘰𝘯 𝘈𝘳𝘤𝘩𝘪𝘵𝘦𝘤𝘵 & 𝘋𝘦𝘷𝘦𝘭𝘰𝘱𝘦𝘳 𝘈𝘴𝘴𝘰𝘤𝘪𝘢𝘵𝘦
𝘊𝘭𝘰𝘶𝘥 𝘛𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺 𝘌𝘯𝘵𝘩𝘶𝘴𝘪𝘢𝘴𝘵