Smart File Organizer Using AWS Lambda & S3 - (Let's Build 🏗️ Series)

Let's say every time a user uploads files to an S3 bucket, you want to move and organize those files in a certain way, meaning if they upload .js I put it inside the /javascript folder, if .txt to the txt folder and so forth.

Is this the use case that you want to build? then this article should be the right one. Alright without further ado let's start with our example.

The main parts of this article:
1- Architecture overview (Terraform)
2- About AWS Services
3- Technical Part (GO code)
4- Result
5- Conclusion

Architecture Overview

In our architecture, we need to make sure that our Lambda function has the right permissions to read and move objects inside the S3 bucket.

📋 Note: If you want to read how to trigger a Lambda function when a new file is uploaded to an S3 bucket, read the following article

Your Lambda function should have the following IAM permissions in order to be able to finish the job successfully.

{
  Version = "2012-10-17",
  Statement = [
  {
    Effect = "Allow",
    Action = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ],
    Resource = "${aws_s3_bucket.image_bucket.arn}/*"
  }
  ]
}

This is our Terraform Lambda resource section to be able to deploy our function.

resource "aws_lambda_function" "file_organizer_lambda" {
  filename         = "./bootstrap.zip"
  function_name    = "lets-build-function"
  handler          = "main"
  runtime          = "provided.al2"
  role             = aws_iam_role.lambda_execution_role.arn
  memory_size      = "128"
  timeout          = "3"
  source_code_hash = filebase64sha256("./bootstrap.zip")
  environment {
    variables = {
      REGION = "${var.region}"
    }
  }
}

About AWS Services

• Amazon S3 will be the service that will store all our files and folders.
• AWS Lambda where we will put our simple/smart peace of GO code to the needed job.

Technical Part (GO code)

Alright before writing our fancy code, let's build the idea in our mind first.

So every time the user uploads a new file, we need to trigger a Lambda function, which will check the file extension and based on that it will copy to a certain folder using AWS SDK.

Here is a simple flow diagram to understand better our functionality.

package main

import (
    "context"
    "fmt"
    "path/filepath"
    "strings"

    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

var (
    svc      = s3.New(session.Must(session.NewSession()))
    bucket   = "YOUR_BUCKET_NAME"
)

func generateTargetKey(record map[string]interface{}) (string, string, string) {
    s3Info := record["s3"].(map[string]interface{})
    bucketName := s3Info["bucket"].(map[string]interface{})["name"].(string)
    objectKey := s3Info["object"].(map[string]interface{})["key"].(string)

    ext := strings.ToLower(filepath.Ext(objectKey))

    var folder string
    switch ext {
    case ".js":
        folder = "javascript/"
    case ".txt":
        folder = "txt/"
    default:
        folder = "others/"
    }

    targetKey := folder + filepath.Base(objectKey)
    return bucketName, objectKey, targetKey
}

func handler(ctx context.Context, event map[string]interface{}) (string, error) {
    records := event["Records"].([]interface{})
    for _, record := range records {
        bucketName, objectKey, targetKey := generateTargetKey(record.(map[string]interface{}))

        if objectKey == targetKey {
            fmt.Printf("Skipping copy for %s as it's already in the correct folder.\n", objectKey)
            continue
        }

        _, err := svc.CopyObject(&s3.CopyObjectInput{
            Bucket:     aws.String(bucketName),
            CopySource: aws.String(bucketName + "/" + objectKey),
            Key:        aws.String(targetKey),
        })
        if err != nil {
            return "", fmt.Errorf("failed to copy object: %v", err)
        }

        _, err = svc.DeleteObject(&s3.DeleteObjectInput{
            Bucket: aws.String(bucketName),
            Key:    aws.String(objectKey),
        })
        if err != nil {
            return "", fmt.Errorf("failed to delete original object: %v", err)
        }

        fmt.Printf("File %s moved to %s\n", objectKey, targetKey)
    }
    return "Files processed successfully", nil
}

func main() {
    lambda.Start(handler)
}

📋 Note: Make sure to save the params in your terminal by running the following, set GOARCH=arm64 & set GOOS=linux and to be sure if things are good you can print the value using this cmd echo %GOARCH%

And run the following cmd:

go build -o bootstrap main.go

For Go functions that use the provided.al2 or provided.al2023 runtime in a .zip deployment package, the executable file that contains your function code must be named bootstrap. If you're deploying the function with a .zip file, the bootstrap file must be at the root of the .zip file.

Result

That's it. Let's upload a couple of files to our S3 bucket and see if they are organized as we want.

After uploading we can see the file was moved to the right folder.

Conclusion

You can use such automation in many interesting features and useful applications for your daily tasks. I hope this was a good and beneficial one for you.

Happy coding.

If you'd like more content like this, please connect with me on LinkedIn.