Use Lambda and DynamoDB to resize S3 Image Uploaded using TERRAFORM

Day 003 - 100DaysAWSIaCDevopsChallenge : Part 1

---

In this article, I am going to create a cloud architecture that allow me to resize and save in DynamoDB tables all objects of type image uploaded inside S3 Bucket following these steps:

• Emit an event after all actions of type s3:ObjectCreated:Put on the S3 Bucket
• A Lambda function captures the above event and then processes it
• The Lambda function get the original object created by its key
• If the object is an image file (with an extension png, jpeg, jpg, bmp, webp or gif), resize the original image using Sharp lib docs
• Store the orginal and resized images in the DynamoDB tables.
• Finally store the resized image in another Bucket

All the steps will be achived using Terraform infrastructure as code.

Architecture Diagram

Beautifull heinnn 😎🤩!? I used cloudairy chart to design it.

Create S3 buckets

The event attached to the bucket will be directed to a Lambda Function. To create S3 Event for Lambda function, we first need to create the Bucket. To do this, create a file named main.tf where all our insfrastructre will be coded. Next, let's create our buckets using terraform:

resource "aws_s3_bucket" "pictures" {
    object_lock_enabled = false
    bucket              = "pictures-<{hash}>"
    force_destroy       = true
    tags = {
        Name = "PicturesBucket"
    }
}

resource "aws_s3_bucket" "thumbs" {
  object_lock_enabled = false
  bucket              = "pictures-<{hash}>-thumbs"
  force_destroy       = true
  tags = {
    Name = "PicturesThumbsBucket"
  }
}

The object_lock_enabled parameter indicates whether this bucket has an Object Lock configuration enabled, it applies only to news resources.
The force_destroy paramater specifies that all objects should be deleted from the bucket when the buckets is destroyed to avoid errors during the destruction proccess (terraform destroy -target=aws_s3_bucket.<bucket_resource_name>).

Now that the bucket is created 🙂, let's attach a trigger to it that will notify the Lambda Function when new object is uploaded to the bucket.

resource "aws_s3_bucket_notification" "object_created_event" {
  bucket = aws_s3_bucket.pictures-bucket.id
  lambda_function {
    events = ["s3:ObjectCreated:*"]
    lambda_function_arn = aws_lambda_function.performing_images_function.arn
  }
  depends_on = [aws_lambda_function.performing_images_function]
}

Note that performing_images_function is the our Lambda function that will be created later in the function section, and aws_s3_bucket.pictures.id is the bucket previously created.

⚠️ Note: As mentionned in the AWS Docs, an S3 Bucket support only one notification configuration. To bypass this issue, I suggest you if you have more that one notification (Lambda invokation, SNS topic trigger, etc.), create one Lambda notification and inside the Lambda function, dispatch your information to others resources (such as other Lambda, SQS,SNS, etc.).

Block public access

The make the buckt publicly inaccessible, there is another terraform resource named aws_s3_bucket_public_access_block to create to achieve this:

resource "aws_s3_bucket_public_access_block" "private_access" {
  bucket                  = aws_s3_bucket.pictures.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Lambda function

Now that the bucket is created and event notification trigger is properly configured, let's create our Lambda function to catch all messages emitted by the bucket. The function code will perform the following operations:

• Retrieve object created - The first operation for our Lambda will be to retrieve the created object if it is of image.
• Resizing the image - Use sharp library to create a miniature (thumb) of the original object.
• Upload resized image to s3 bucket dedicated - Upload the thumb image to another s3 bucket.
• Save the original and resized image metadata - After the image is resized without error, the metadata such as URL, Object key, size, etc., will be stored in two dynamoDB tables: one for original image and another for the resized image.

Before creating the Lambda function, we need to grant it the necessaries permissions to interact with others resources and vice versa:

• Allow the bucket to Invoke function - lambda:InvokeFunction
• Allow the Lambda function to get objects inside the bucket - s3:GetObject and s3:GetObjectAcl
• Allow the Lambda function to put items in the dynamodb tables - dynamodb:PutItem.

Lambda Assume Role

Generate an IAM policy that allow action sts:AssumeRole where identifier of type Service is lambda.amazonaws.com.

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"
    principals {
      identifiers = ["lambda.amazonaws.com"]
      type = "Service"
    }
    actions = ["sts:AssumeRole"]
  }
}

Now attach the assume_role policy to our future Lambda resource role.

resource "aws_iam_role" "for_lambda" {
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
  name               = "iam_role_for_lambda"
  tags = {
    Name = "IAM:Role:Lambda"
  }
}

Create IAM Policy for lambda

Let's create a new policy to allow Lambda to:

• get S3 objects
• create S3 objects
• put items into DynamoDB tables.

resource "aws_iam_policy" "lambda_policy_for_s3_and_dyanmodb" {
  name        = "lambda-create-object-and-put-item_policy"
  description = "The IAM policy to allow Lambda to get S3 objects, put objects in S3, and put items in DynamoDB tables"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:PutObjectAcl"
        ],
        Resource = [
          "${aws_s3_bucket.thumbs.arn}/*", # will be created later
        ]
      },
      {
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:GetBucket"]
        Resource = [
          "${aws_s3_bucket.pictures.arn}/*"
        ]
      },
      {
        Effect = "Allow"
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action : ["dynamodb:PutItem"]
        Resource : [
          aws_dynamodb_table.pictures.arn, # will be created later
          aws_dynamodb_table.thumbnails.arn # will be created later
        ]
      }
    ]
  })
  path = "/"
  tags = {
    Name = "iam:policy:lambda-for-s3-and-dynamodb"
  }
}

In the policy, we have also allowed Lambda to log its activities into CloudWatch, which will permit us to visualize all activities inside the Lambda as shown below:

Action = [
    "logs:CreateLogGroup",
    "logs:CreateLogStream",
    "logs:PutLogEvents"
]

Attach Lambda policy to the Lambda role

resource "aws_iam_role_policy_attachment" "attach_policies_to_lambda_role" {
  policy_arn = aws_iam_policy.lambda_policy_for_s3_and_dyanmodb.arn
  role       = aws_iam_role.for_lambda.name
}

And the waiting time over !!! 🧘🏾‍♂️ let's jump into Lambda creation

Create the lambda function

Before creating the terraform Lambda resource we need first to write code that will be executed inside the function.
Create the files index.ts ans package.json inside assets/lambda directory in the root project. Below is the content the assets/lambda/package.json:

{
  "main": "index.js",
  "type": "module",
  "scripts": {},
  "dependencies": {
    "@aws-sdk/client-dynamodb": "^3.609.0",
    "@aws-sdk/client-s3": "^3.609.0",
    "@aws-sdk/lib-dynamodb": "^3.610.0",
    "sharp": "^0.33.4",
    "uuid": "^10.0.0"
  }
}

and run npm install inside the assets/lambda directory to install dependencies. ⚠️ the node_modules is fondamental for the function to execute properly. (in one of my future article I will show you how to optimize it by using node_modules in the layers if you want to launch more than on function for more optimization)

cd assets/lambda
npm install

The function source code assets/lambda/index.ts:

import {GetObjectCommand, PutObjectCommand, S3Client} from "@aws-sdk/client-s3";
import sharp from "sharp";
import {DynamoDBClient} from "@aws-sdk/client-dynamodb";
import {DynamoDBDocumentClient, PutCommand} from "@aws-sdk/lib-dynamodb";
import {v4 as UUID} from "uuid";

const region = process.env.REGION;
const thumbsDestBucket = process.env.THUMBS_BUCKET_NAME;
const picturesTableName = process.env.DYNAMODB_PICTURES_TABLE_NAME;
const thumbnailsTableName = process.env.DYNAMODB_THUMBNAILS_PICTURES_TABLE_NAME;
const s3Client = new S3Client({
    reqion: region,
});

const dynClient = new DynamoDBClient({
    region: region,
});
const documentClient = DynamoDBDocumentClient.from(dynClient);

export const handler = async (event, context) => {
    const bucket = event.Records[0].s3.bucket.name;
    const objKey = decodeURIComponent(event.Records[0].s3.object.key.replace("/\+/g", " "));
    if(new RegExp("[\/.](jpeg|png|jpg|gif|svg|webp|bmp)$").test(objKey)) {
        try {
            const originalObject = await s3Client.send(new GetObjectCommand({
                Bucket: bucket,
                Key: objKey
            }));
            console.log("Get S3 Object: [OK]");
            const imageBody = await originalObject.Body.transformToByteArray();
            const thumbs = await sharp(imageBody)
                .resize(128)
                .png()
                .toBuffer();
            console.log("Image resized: [OK]");
            await s3Client.send(new PutObjectCommand({
                Bucket: thumbsDestBucket,
                Key: objKey,
                Body: thumbs
            }));
            console.log("Put resized image into S3 bucket: [OK]");
            const itemPictureCommand = new PutCommand({
                TableName: picturesTableName,
                Item: {
                    ID: UUID(),
                    ObjectKey: objKey,
                    BucketName: bucket,
                    Region: region,
                    CreatedAt: Math.floor((new Date().getTime()/1000)),
                    FileSize: event.Records[0].s3.object.size
                }
            });

            await documentClient.send(itemPictureCommand);

            console.log("Put original metadata into DynamoDB Table: [OK]");

            const itemThumbCommand = new PutCommand({
                TableName: thumbnailsTableName,
                Item: {
                    ID: UUID(),
                    ObjectKey: objKey,
                    BucketName: thumbsDestBucket,
                    Region: region,
                    CreatedAt: Math.floor((new Date().getTime()/1000)),
                    FileSize: thumbs.byteLength
                }
            });

            await documentClient.send(itemThumbCommand);
            console.log("Put resized metadata into DynamoDB Table: [OK]");
            console.debug({
                statusCode: 200,
                body: JSON.stringify({
                    object: `${bucket}/${objKey}`,
                    thumbs: `${thumbsDestBucket}/${objKey}`
                })
            })
        } catch (e) {
            console.error(e);
            console.debug({
                statusCode: 500,
                body: JSON.stringify(e)
            });
        }
    }
};

Return to the Terraform. Now that the source code is ready, we can now create our Terraform function resource.
Once again, we need to zip our source code, the entire assets/lambda directory including node_modules. To zip the function source code, we will use Terraform archive_file resource like this:

data "archive_file" "function" {
  output_path = "./assets/func.zip"
  type        = "zip"
  source_dir  = "./assets/lambda"
}

We have zipped the entire content of assets/lambda to assets/func.zip file.

And Terraform function resource:

data "aws_region" "current" {}

resource "aws_lambda_function" "performing_images_function" {
  function_name    = "performing-images-function"
  role             = aws_iam_role.for_lambda.arn
  handler          = "index.handler"
  runtime          = "nodejs18.x"
  filename         = "./assets/func.zip"
  source_code_hash = data.archive_file.function.output_base64sha256
  memory_size      = 128
  timeout          = 10

  timeouts {
    create = "30m"
    update = "40m"
    delete = "40m"
  }

  environment {
    variables = {
      TRIGGER_BUCKET_NAME                     = aws_s3_bucket.pictures-bucket.bucket
      THUMBS_BUCKET_NAME                      = aws_s3_bucket.thumbs.bucket
      REGION                                  = data.aws_region.current.name
      DYANMODB_THUMBNAILS_PICTURES_TABLE_NAME = aws_dynamodb_table.thumbnails.name
      DYANMODB_PICTURES_TABLE_NAME            = aws_dynamodb_table.pictures.name
    }
  }

  depends_on = [data.archive_file.function]

  tags = {
    Name = "Lambda:PerformingImages"
  }
}

⚠️⚠️ Note: the parameter source_code_hash is important because, if the code changes, it signals Terraform to update Lambda function with the new content.

⚠️ Also it is important to zip source before creating the function, as indicated by the line:

depends_on = [data.archive_file.function]

And the last Terraform resource and the must important one in our Lambda section, is aws_lambda_permission, that grants permission to S3 Bucket to invoke Lambda for all object-created events:

resource "aws_lambda_permission" "allow_bucket" {
  statement_id  = "AllowExecutionFromBucket"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.performing_images_function.arn
  source_arn    = aws_s3_bucket.pictures.arn
  principal     = "s3.amazonaws.com"
}

Create DynamoDB tables

We are now going to create two DynamoDB tables to persist the information about the original object and the resized image. As lambda function is already configured with dynamodb:PutItem, let's define those tables:

resource "aws_dynamodb_table" "pictures" {
  name         = "PictureTable"
  table_class  = "STANDARD"
  hash_key     = "ID"
  range_key    = "ObjectKey"
  billing_mode = "PAY_PER_REQUEST"
  dynamic "attribute" {
    for_each = local.dynamo_table_attrs
    content {
      name = attribute.key
      type = attribute.value
    }
  }
  tags = {
    Name = "dynamodb:PictureTable"
  }
}

resource "aws_dynamodb_table" "thumbnails" {
  name         = "ThumbnailsTable"
  table_class  = "STANDARD"
  hash_key     = "ID"
  range_key    = "ObjectKey"
  billing_mode = "PAY_PER_REQUEST"
  dynamic "attribute" {
    for_each = local.dynamo_table_attrs
    content {
      name = attribute.key
      type = attribute.value
    }
  }
  tags = {
    Name = "dynamodb:ThumbnailsTable"
  }
}

🥳✨woohaah!!!
We have reached the end of the article.
Thank you so much 🙂

---

Your can find the full source code on GitHub Repo

Feel free to leave a comment if you need more clarification or if you encounter any issues during the execution of your code.