Filtering DynamoDB event for AWS Lambda functions

Filtering DynamoDB event for AWS Lambda functions

AWS Lambda now provides content filtering options DynamoDB as event sources. With this, you can filter the event to only include the data you want. This helps reduce traffic to customers’ Lambda functions, simplifies code, and reduces overall cost.

But AWS CDK is not yet able to support this. You can track the progress of this feature in the AWS CDK Lambda event filtering

How to filter DynamoDB event for AWS Lambda functions in CDK

Let's look at an example of how to filter DynamoDB event for AWS Lambda functions in CDK.

// 1. Create a DynamoDB table
const dynamoTable = new dynamodb.Table(this, 'DynamoTable', {
  partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
  removalPolicy: RemovalPolicy.DESTROY,
  stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
})

// 2. Create a Lambda function that will be triggered by the DynamoDB stream
// you can see how to create lambda functions in typescript in the following link:
// https://www.codewithyou.com/blog/writing-typescript-lambda-in-aws-cdk
const fn = new NodejsFunction(this, 'Handler', {
  entry: './lambda/index.ts',
  handler: 'main',
  runtime: lambda.Runtime.NODEJS_14_X,
  architecture: lambda.Architecture.ARM_64,
  bundling: {
    externalModules: ['aws_sdk'],
    minify: true,
  },
})

// 3. currently, aws cdk is not yet support filtering event sources
// https://github.com/aws/aws-cdk/issues/17874
// So, we need create a source mapping to map the DynamoDB table to the Lambda function
const sourceMapping = new lambda.EventSourceMapping(this, 'SourceMapping', {
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  batchSize: 1, // only process one record at a time
  target: fn, // the lambda function to trigger
  eventSourceArn: dynamoTable.tableStreamArn, // the DynamoDB stream to read from
  bisectBatchOnError: true, // if the Lambda function fails, split the batch into two
  retryAttempts: 3, // retry 3 times if the Lambda function fails
})

// grant the function stream access to the DynamoDB table
dynamoTable.grantStreamRead(fn)

// filter the event source mapping to only process records with the specified event type (in this case, "INSERT")
const cfnSourceMapping = sourceMapping.node.defaultChild as lambda.CfnEventSourceMapping // get the underlying CloudFormation resource

cfnSourceMapping.addPropertyOverride('FilterCriteria', {
  Filters: [
    {
      Pattern: JSON.stringify({
        eventName: ['INSERT'], // only process INSERT events
      }),
    },
  ],
})

Let's go through the code above.

1. We created a DynamoDB table. This table will be the source of the event.
2. We created a Lambda function written in TypeScript. You can see how to create lambda functions in TypeScript in the following link: https://www.codewithyou.com/blog/writing-typescript-lambda-in-aws-cdk
3. We created a source mapping to map the DynamoDB table to the Lambda function. This source mapping will be the event source for the Lambda function.
4. We filtered the source mapping to only process records with the specified event type (in this case, "INSERT").

cfnSourceMapping.addPropertyOverride('FilterCriteria', {
  Filters: [
    {
      Pattern: JSON.stringify({
        eventName: ['INSERT'], // only process INSERT events
      }),
    },
  ],
})

This code snippet will add a filter to the source mapping. The filter will only process records with the event type "INSERT". The filter will be added to the source mapping as a property override.

The code for this article is available on GitHub

Let deploy the whole stack to AWS.

yarn deploy

After the stack is deployed, you can see the Lambda function is triggered by the DynamoDB stream by add a item to the DynamoDB table.

Clean up

Don't forget to delete the stack. You can do this by running the following command:

npx cdk destroy

Thanks for reading this article. If you have any questions, please leave a comment. I will try to answer your questions as soon as possible.