Creating a Lambda Layer Using AWS CodeBuild and Terraform

AWS Lambda layers allow you to share common dependencies across multiple Lambda functions, reducing redundancy and improving maintainability. In this blog post, we will create a Lambda layer for the Python requests library using AWS CodeBuild and automate the process using Terraform.

Using AWS CodeBuild to create a Lambda layer has several advantages over manually creating it on a local machine and uploading it:

1. CodeBuild ensures a standardized environment, reducing inconsistencies caused by different local setups.
2. CodeBuild can be integrated into CI/CD pipelines, allowing automatic layer updates when dependencies change.
3. Using CodeBuild reduces the need to store and transfer dependencies manually, minimizing security risks.
4. Developers do not need to install dependencies locally, avoiding conflicts with their existing Python setup.

Step 1: Write a buildspec.yml file

The buildspec.yml file defines the build process for AWS CodeBuild. Let's break it down:
phases:
install: Sets the runtime environment, ensuring Python 3.12 is available.
build: Executes the commands to create a directory for dependencies, install the requests library inside it, and package everything into a ZIP file.
post_build: Create a lambda layer using CLI.
artifacts: Defines which files should be saved after the build is complete (i.e., requests_layer.zip).

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.12
  build:
    commands:
      - mkdir python
      - cd python
      - |
        cat <<'EOF' > requirements.txt
        requests
        EOF
      - pip install --platform=manylinux_2_17_x86_64 --only-binary=":all:" -r requirements.txt -t .
      - cd ..
      - zip -r9 $LAYER_NAME.zip python
  post_build:
    commands:
      - test "$CODEBUILD_BUILD_SUCCEEDING" = "1"
      - aws lambda publish-layer-version --layer-name $LAYER_NAME --description "Example Lambda layer" --compatible-runtimes python3.12 --zip-file fileb://${LAYER_NAME}.zip

Step 2: Create required IAM Role and attach Policy

Define an IAM Role called ExampleLambdaLayerCodeBuildRole and a policy which allows AWS CodeBuild (codebuild.amazonaws.com) to assume this role using sts:AssumeRole.

################################################################################
# Lambda IAM role to assume the role
################################################################################
resource "aws_iam_role" "codebuild_role" {
  name = "CodeBuildRole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "codebuild.amazonaws.com"
        }
      }
    ]
  })
}

IAM Policy Attached to the Role
CloudWatch Logs Permissions: Allows CodeBuild to create log groups, streams, and publish logs.
Lambda Layer Permissions: Allows lambda:PublishLayerVersion, which lets CodeBuild publish new Lambda layers.

################################################################################
# Assign policy to the role
################################################################################
resource "aws_iam_role_policy" "codebuild_policy" {
  name = "CodeBuildPolicy"
  role = aws_iam_role.codebuild_role.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ]
        Effect   = "Allow"
        Resource = "arn:aws:logs:${var.aws_region}:${local.account_id}:log-group:/aws/codebuild/*"
      },
      {
        Action = [
          "lambda:PublishLayerVersion"
        ],
        Effect   = "Allow"
        Resource = "arn:aws:lambda:${var.aws_region}:${local.account_id}:layer:*"
      }
    ]
  })
}

Step 3: Create a CodeBuild Project Using Terraform

resource "aws_codebuild_project" "requests_layer_build" {
  name         = "requests_layer_build"
  description  = "Build project for requests_layer"
  service_role = aws_iam_role.codebuild_role.arn
  artifacts {
    type = "NO_ARTIFACTS"
  }
  environment {
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/amazonlinux2-x86_64-standard:5.0"
    type                        = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"
    environment_variable {
      name  = "LAYER_NAME"
      value = "requests_layer"
    }
  }
  source {
    buildspec = file("${path.module}/buildspec.yml")
    type      = "NO_SOURCE"
  }
}

Step 4: Start the CodeBuild Project Manually

You can manually start the build from the AWS Console or trigger it using the AWS CLI:

aws codebuild start-build --project-name lambda-layer-builder

Steps to Run Terraform

Follow these steps to execute the Terraform configuration:

terraform init
terraform plan 
terraform apply -auto-approve

Upon successful completion, Terraform will provide relevant outputs.

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Testing

CodeBuild Project

IAM Role with policies

CodeBuild manual start of the project

[Container] 2025/02/11 12:09:27.671410 Running on CodeBuild On-demand
[Container] 2025/02/11 12:09:27.671422 Waiting for agent ping
[Container] 2025/02/11 12:09:27.973022 Waiting for DOWNLOAD_SOURCE
[Container] 2025/02/11 12:09:28.419982 Phase is DOWNLOAD_SOURCE
[Container] 2025/02/11 12:09:28.526637 CODEBUILD_SRC_DIR=/codebuild/output/src3724790750/src
[Container] 2025/02/11 12:09:28.526735 YAML location is /codebuild/readonly/buildspec.yml
[Container] 2025/02/11 12:09:28.526870 No commands found for phase name: install
[Container] 2025/02/11 12:09:28.529047 Processing environment variables
[Container] 2025/02/11 12:09:28.848570 Selecting 'python' runtime version '3.12' based on manual selections...
[Container] 2025/02/11 12:09:28.849133 Running command echo "Installing Python version 3.12 ..."
Installing Python version 3.12 ...

[Container] 2025/02/11 12:09:28.854234 Running command pyenv global $PYTHON_312_VERSION

[Container] 2025/02/11 12:09:29.948251 Moving to directory /codebuild/output/src3724790750/src
[Container] 2025/02/11 12:09:29.948277 Cache is not defined in the buildspec
[Container] 2025/02/11 12:09:29.948283 Cache is not defined in the buildspec
[Container] 2025/02/11 12:09:29.951334 Unable to initialize cache download: no paths specified to be cached
[Container] 2025/02/11 12:09:30.032189 Configuring ssm agent with target id: codebuild:ac95fd48-f829-481c-8138-5407fc80ac54
[Container] 2025/02/11 12:09:30.071707 Successfully updated ssm agent configuration
[Container] 2025/02/11 12:09:30.072044 Registering with agent
[Container] 2025/02/11 12:09:30.159585 Phases found in YAML: 3
[Container] 2025/02/11 12:09:30.159605  BUILD: 6 commands
[Container] 2025/02/11 12:09:30.159610  POST_BUILD: 2 commands
[Container] 2025/02/11 12:09:30.159614  INSTALL: 0 commands
[Container] 2025/02/11 12:09:30.159843 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
[Container] 2025/02/11 12:09:30.159856 Phase context status code:  Message: 
[Container] 2025/02/11 12:09:30.276868 Entering phase INSTALL
[Container] 2025/02/11 12:09:30.450419 Phase complete: INSTALL State: SUCCEEDED
[Container] 2025/02/11 12:09:30.450439 Phase context status code:  Message: 
[Container] 2025/02/11 12:09:30.484673 Entering phase PRE_BUILD
[Container] 2025/02/11 12:09:30.486503 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2025/02/11 12:09:30.486518 Phase context status code:  Message: 
[Container] 2025/02/11 12:09:30.516317 Entering phase BUILD
[Container] 2025/02/11 12:09:30.517605 Running command mkdir python

[Container] 2025/02/11 12:09:30.524225 Running command cd python

[Container] 2025/02/11 12:09:30.529521 Running command cat <<'EOF' > requirements.txt
requests
EOF


[Container] 2025/02/11 12:09:30.535347 Running command pip install --platform=manylinux_2_17_x86_64 --only-binary=":all:" -r requirements.txt -t .
Collecting requests (from -r requirements.txt (line 1))
  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests->-r requirements.txt (line 1))
  Downloading charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (35 kB)
Collecting idna<4,>=2.5 (from requests->-r requirements.txt (line 1))
  Downloading idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting urllib3<3,>=1.21.1 (from requests->-r requirements.txt (line 1))
  Downloading urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 1))
  Downloading certifi-2025.1.31-py3-none-any.whl.metadata (2.5 kB)
Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Downloading certifi-2025.1.31-py3-none-any.whl (166 kB)
Downloading charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (145 kB)
Downloading idna-3.10-py3-none-any.whl (70 kB)
Downloading urllib3-2.3.0-py3-none-any.whl (128 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2025.1.31 charset-normalizer-3.4.1 idna-3.10 requests-2.32.3 urllib3-2.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip

[Container] 2025/02/11 12:09:47.941137 Running command cd ..

[Container] 2025/02/11 12:09:47.946567 Running command zip -r9 $LAYER_NAME.zip python
.
.
.
.
.

[Container] 2025/02/11 12:09:48.379476 Phase complete: BUILD State: SUCCEEDED
[Container] 2025/02/11 12:09:48.379508 Phase context status code:  Message: 
[Container] 2025/02/11 12:09:48.414851 Entering phase POST_BUILD
[Container] 2025/02/11 12:09:48.415965 Running command test "$CODEBUILD_BUILD_SUCCEEDING" = "1"

[Container] 2025/02/11 12:09:48.421336 Running command aws lambda publish-layer-version --layer-name $LAYER_NAME --description "Example Lambda layer" --compatible-runtimes python3.12 --zip-file fileb://${LAYER_NAME}.zip
{
    "Content": {
        "Location": "https://prod-iad-c1-djusa-layers.s3.us-east-1.amazonaws.com/snapshots/197317184204/requests_layer-92f2e283-b941-4bc5-a883-36f57acc6cc3?versionId=_e9Kd2pPsBUIipHsDsJmDo8lQG9dUhy4&X-Amz-Security-Token=IQoJb3JpZ2luX2VjELz%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDc9Z7BPB5nFMfFB%2BBWCME2%2Fd%2FcBSbDO7zznBCThWYEKQIhAP7bpVW7CmF2SBzmubwzCzIg%2Fwiy8m2svH5evbRcJVchKsMFCNX%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQBBoMNDc5MjMzMDI1Mzc5Igw7VQuxi29S7dAnM2gqlwVgYRDuuEh7hm3y1dNzwn5SsHoMm%2B%2BFLNXjvlPQjg1v%2B2gDW2U2WpMXS5nNDf%2BceBHKf6ujvwuOHk93VrX0vLuGaqF8967e%2BS6e%2F1K1XZJP3uErktjhEXqA0FyzUDlOL6y09N881oftoi9821yu3ryojC3RlIRl8dzl2LJWcOXC8ApY1ntqDcAOuR6Atr3SB4Z3Vka5CvvM4MtZD09YiJSFXt0UHVnP1FZaHJt9Q14JHictFwaA%2BFYzh8ko3sNJ3SsAyOztTtSMXUZemE0f2onnUPLfLU57Tk2QnCggN2HhSni1ZcfZXxVqUAWOCo9X6tJhpRTklUaSx6H53Hvcj3%2FPWoH%2FGYifNiWT4sXZZEn%2Fw2sM9EdrqzKCpl1EbqbCCh7c5%2BQaDQeNqhJXF12IV%2Fj1JFZ4EWnK0bHifaaz5vwowgeUvinrXBncErB37BOA2CA5bFQK%2BgCUm4jeryKX%2BpZaClVO4N9ldUagJHAHxt6LKvs4KUGvXV%2FDUc3T6eMXrWq9O8reTbwvJFgWQMFQp7nvC4vMhiKZ74MigqGoPBJOn4jRIYMIXWNMY0FdkoCuBgcWTLI%2BY0w5iV4o96VoZS10UmAv%2BOSdd7fSWpmcaF%2Bv0kw%2BZEL1z%2BL793jZ%2B3Yykx8LJZqlv7cBfa1gJLVTrF6mRmlXkm7ZezaexmdPC2JHe4oQMrv6SZpZHFi8jBOmcAMuBRXtVXAYtn3VVhrjKY%2B%2BZ8Wox333U42cuDKZxvOuzMxWAtwYow%2BiZqCozWYqEL%2B798NQXmcaEUJ4uWU1D36JqXq3jHZP5Xz9cQb69o1hLkXyKpyXfdv%2BfhzpUdBNZfXEfrhp3VjNkvcEtc00SwBe7QFJ8ROlqH5BQBfTKSiioIgLVg4GnGwwjuisvQY6sAFdKea5Uoe8WzRZznygvaplsDUSNMmRxDe7%2F0Z%2FTFqosJcvGOayN1aDabMoGU%2Ft%2BcuqRTIgvxeMFk0NjxaX%2Fn5gyqgQ%2F0ZNj4ZJsuuUTLEHT85%2F%2BD1oIJ%2FaK0pkqbyXfwrTKC%2FtkJli2ga%2FayGxJUUqRxvW32rWncdKtWlMHasOqEVU0TRrV6Qg8GcHXmGa3iwJ6rw7tBRm6cpZlDN5VpSQ3%2FDh5hS9KrvhUTdWo87lww%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250211T121000Z&X-Amz-SignedHeaders=host&X-Amz-Expires=600&X-Amz-Credential=ASIAW7FEDUVR6EHRB4HR%2F20250211%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=680fe28241f05dd35be5d439954149f60389fef4d94e74d11398915eb8e1a728",
        "CodeSha256": "/Suk2JbqGwoZNe5tE6zPVCMgaexrfgD3BqQLsCuVS1E=",
        "CodeSize": 1048421
    },
    "LayerArn": "arn:aws:lambda:us-east-1:197317184204:layer:requests_layer",
    "LayerVersionArn": "arn:aws:lambda:us-east-1:197317184204:layer:requests_layer:5",
    "Description": "Example Lambda layer",
    "CreatedDate": "2025-02-11T12:10:01.417+0000",
    "Version": 5,
    "CompatibleRuntimes": [
        "python3.12"
    ]
}

[Container] 2025/02/11 12:10:01.544143 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2025/02/11 12:10:01.544160 Phase context status code:  Message: 
[Container] 2025/02/11 12:10:01.605342 Set report auto-discover timeout to 5 seconds
[Container] 2025/02/11 12:10:01.605399 Expanding base directory path:  .
[Container] 2025/02/11 12:10:01.608418 Assembling file list
[Container] 2025/02/11 12:10:01.608434 Expanding .
[Container] 2025/02/11 12:10:01.611503 Expanding file paths for base directory .
[Container] 2025/02/11 12:10:01.611517 Assembling file list
[Container] 2025/02/11 12:10:01.611521 Expanding **/*
[Container] 2025/02/11 12:10:01.615443 No matching auto-discover report paths found
[Container] 2025/02/11 12:10:01.615461 Report auto-discover file discovery took 0.010119 seconds
[Container] 2025/02/11 12:10:01.615483 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
[Container] 2025/02/11 12:10:01.615490 Phase context status code:  Message: 

Build History

Lambda Layer

Cleanup

Remember to stop AWS components to avoid large bills.

terraform destroy -auto-approve

Delete the lambda layer manually if you are not going to use it!

Conclusion

By following these steps, we have successfully automated the creation of a Lambda layer for the Python requests library using AWS CodeBuild and Terraform. This approach ensures that our dependencies are packaged consistently and can be reused across multiple Lambda functions, improving efficiency and maintainability.

References

1. GitHub Repo https://github.com/chinmayto/terraform-aws-codebuild-lambda-layer