Python src Layout for AWS Lambdas

Over the last several years, the Python community has been moving towards a packaging layout format known as the src layout as the recommended way to organize directories and files for Python Packages.

The main takeaways of the src layout are:

• Your project has a src folder
  • The src folder only has sub folder[s]. Each sub folder
  • Is a Python Package
  • Its name is the name of the package
  • It contains an __init__.py (that's pretty much what makes it a Python Package)
  • All Python source code that makes up your appplication (other than tests or build utilities) for the project are in these sub folder[s]
• Modern Python tooling expects / works best with this layout
• Makes packaging easier and less prone to errors
• "editable" installs for development and regular installation for testing
• Generally no need for "import trickery"
• References at end of this article

We were looking for standards and best practices for organizing our Python AWS Lambda projects using Poetry in our monorepo and kept coming across the src layout as the recommended way to organize Python projects that had multiple modules. At the same time just about all examples of Python Lambda projects did not use src layout and in fact seemed to want to have the handler module at the top level of the project directory. There just weren't many examples of this combination of src layout Poetry and AWS Lambdas.

Turns out that it wasn't really difficult. The main "trick" is setting the Lambda handler name correctly.

All this example code is available at Informed/blogpost-python-src-layout

Example src layout for an AWS Lambda

Here's an example src layout suitable for AWS Lambda

• This is a section of what could be a regular or monorepo that shows one lambda project: my_lambda
  • You could have more lambda projects under services, each would be a separate subproject with the same layout
  • How to implement such a multi-project monorepo is left to a future blog post
  • The actual lambda code is in src/my_lambda
  • There is an example of additional non-code files in src/my_lambda/stuff that might be used by the lambda or lambda layers such as an otel layer that will be part of the deployed zip image.
    • This is not required, just showing an example of how you can easily bundle in additional non-code files
  • You can have additional Python modules in here like utils.py that are imported into the handler.py module
  • If your Lambda application is more complex you can have local Packages as subdirectories of src/my_lambda

blogpost-python-src-layout
├── LICENSE
├── README.md
├── doc
│   └── python_src_layout_for_aws_lambdas.md
└── services
    └── my_lambda
        ├── CHANGELOG.md
        ├── LICENSE
        ├── README.md
        ├── poetry.lock
        ├── pyproject.toml
        ├── src
        │   └── my_lambda
        │       ├── __init__.py
        │       ├── handler.py
        │       ├── stuff
        │       │   ├── config.yml
        │       │   └── data.csv
        │       └── utils.py
        └── tests
            └── test_my_handler.py

Example Poetry pyproject.toml

[tool.poetry]
name = "my_lambda"
version = "0.0.0"
description = "This is my lambda which is mine"
authors = ["Da Dev <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.9"
pyyaml = "6.0"

[tool.poetry.group.dev.dependencies]
coverage = {extras = ["toml"], version = "^6.5.0"}
pytest = "^7.2.0"
flake8 = "^6.0.0"
python-lambda-local = "^0.1.13"
boto3 = "1.20.32"

[tool.pytest.ini_options]
addopts = ["--import-mode=importlib"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry expects a src layout. The fact that the name is the same as the directory name under src and that directory is a Python Package (i.e. it has an __init__.py and source code) means that Poetry will treat everything in src/my_lambda as the package it's going to build. Everything in src/my_lambda and any dependencies under [tool.poetry.dependencies] will end up in the deployed zip image that will become the lambda function image. In this example we have the pyyaml package included.

The dependencies under [tool.poetry.group.dev.dependencies] will only be used for local builds, and not included in the zip image. You can put dependencies that are in the lambda runtime, like boto3 or any lambda layers (like Otel, or aws_powertools) in [tool.poetry.group.dev.dependencies] if you want them available for local testing.

There is also the option (and some folks recommend) to put a more up to date and fixed version of boto3 in [tool.poetry.dependencies] since AWS can change the version used in the runtime at any time. The main downside of this is it makes your uploaded image larger.

The section '[tool.pytest.ini_options]' is recommended for all new packages that use pytest for running their tests.

Set the Lambda Handler to be the "two dot" format

The main thing that is different for configuring Lambdas when you use the src layout is that you must specify the handler as

<package_name>.<handler_module_name>.<handler_function_name>`

Also known as the 2 dot solution that specifies the handler like a package import path.

In our example that would be:

my_lambda.handler.handler

Example src/my_lambda/handler.py

This is just a very minimal lambda function that demonstrates:

• Importing the PyYaml package we specify as a dependency in the pyproject.toml
• Printing the event info passed in the handler
• Calling an imported function from the same lambda package (utils.my_util)
• Fetching and printing one of the non-source files in src/my_lambda/stuff

import json
import os
import pkgutil
from . import utils

# We're not using pyyaml, just showing that it's installed
import yaml

print("Loading function")


def handler(event, context):
    print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event["key1"])
    print("value2 = " + event["key2"])
    print("value3 = " + event["key3"])

    print(utils.my_util())
    print(f"cwd: {os.getcwd()}")
    lines = pkgutil.get_data(__name__, "my_lambda/stuff/config.yml")
    print("File contents of my_lambda/stuff/config.yml:")
    print(lines)

    return event["key1"]  # Return the first key value

Building and deploying the Lambda

This is the basics of building and deploying the lambda using Poetry. You may have a more sophisticated CI/CD process.

These commands should be executed in the top level of the project (i.e. my_repo/services/my_lambda)

Building the distribution

Note: These commands may not work for your actual application if it has dependencies that need to be compiled as part of the build process, particularly if you are building your application on a machine that is not the same architecture as your target Lambda (i.e. building on an M1 Macintosh and targeting an x86_64 lambda). In that case you need to use Docker to do your building which is beyond the scope of this post.

These examples were only tested on an M1 Mac but should work on any modern Mac or *nix machine and will build zip images suitable for targeting Linux x86 Lambdas if you do not need to build binary dependencies.
The pip install shown later uses the argument --platform manylinux2014_x86_64 which forces the packaging to only include binary wheels built for Linux x86_64. The argument --only-binary :all: ensures that it will not try to compile any source only dependencies and instead will emit an error letting you know that you must build the package in the native target environment (i.e. use a Docker build process).

1) First we export the dependencies from Poetry so that we can later use pip install to create the files needed for the zip image.

poetry export -f requirements.txt --output requirements.txt  --without-hashes

2) Have Poetry build all the wheels and such

poetry build

This will result in a bunch of files in the directory dist the top level of your project.

3) Use pip to create the packages

poetry run pip install -r requirements.txt --upgrade --only-binary :all: --platform manylinux2014_x86_64 --target package dist/*.whl

This will generate all the wheels of all the dependencies in the package directory, suitable for zipping into the lambda image.

4) Zip up the image

cd package
mkdir -p out
zip -r -q out/my-lambda.zip . -x '*.pyc' out
cd ..

This will result in a file in my_repo/services/my_lambda/package/out/my-lambda.zip that is suitable for uploading to AWS as the Lambda Function image.

Create the Lambda function

You could do this in the AWS console or use the following AWS CLI Commands (If you use the AWS Console, it will create the execution role and trust policy automatically by default).

1) Create the execution role and trust policy

aws iam create-role --role-name my-lambda-ex --assume-role-policy-document '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'

• You should get a result like the following.
• Copy the arn from this output and you will use it for the next command

{
    "Role": {
        "Path": "/",
        "RoleName": "my-lambda-ex",
        "RoleId": "AROAWUWOOBVLDKY7ZE7P3",
        "Arn": "arn:aws:iam::1234567890123:role/my-lambda-ex",
        "CreateDate": "2023-01-12T06:08:22+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

2) Create the Lambda function

• You have to set the handler to the 2 dot style my_lambda.handler.handler

• Change the fake account-id (1234567890123) in the role arn to your AWS account-id
  

  aws lambda create-function --function-name my-lambda \
    --zip-file fileb://package/out/my-lambda.zip \
    --handler my_lambda.handler.handler --runtime python3.9 \
    --role arn:aws:iam::1234567890123:role/my-lambda-ex
  

If you already have created the Lambda function

If you created the Lambda function some other way like via the Console or you want to update the Lambda function you created earlier, you can use the following command just to update it.

This mechanism loads the lambda from the S3 asset you uploaded earlier

aws lambda update-function-code --function-name my-lambda \
  --zip-file fileb://package/out/my-lambda.zip \
  --region us-west-2

Test the Lambda

At this point your lambda should be ready to test. You can use the default test input in the AWS Lambda Console:

{
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}

You can run the default Test on the AWS Lambda console and you will see all the log output as well as the result which would look something like:

Test Event Name
basic

Response
"value1"

Function Logs
Loading function
START RequestId: 2f33f7b2-ddae-46df-a61b-1263ef404d6b Version: $LATEST
Received event: {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
value1 = value1
value2 = value2
value3 = value3
Hello from my_util
cwd: /var/task
File contents of my_lambda/stuff/config.yml:
['thing:\n', '  hand: manicure\n']
END RequestId: 2f33f7b2-ddae-46df-a61b-1263ef404d6b
REPORT RequestId: 2f33f7b2-ddae-46df-a61b-1263ef404d6b  Duration: 1.57 ms   Billed Duration: 2 ms   Memory Size: 128 MB Max Memory Used: 40 MB  Init Duration: 172.83 ms

Or you could run it via the AWS CLI:

aws lambda invoke --cli-binary-format raw-in-base64-out \
  --function-name my-lambda \
  --payload '{ "key1": "value1", "key2": "value2", "key3": "value3"}' \
  outputfile.txt   --log-type Tail \
  --query 'LogResult' --output text |  base64 -d

This will print the log output (same as what was shown above in example of running the Test in the AWS Console)

You can see the returned value of the handler in outputfile.txt

Running tests with Pytest

There is also a simple test example in services/my_lambda/tests/test_my_handler.py

import pytest
import os
from my_lambda.handler import handler

event = {"key1": "value1", "key2": "value2", "key3": "value3"}
context = {}


def test_my_handler():
    # Emulate running in the same directory context as the lambda would
    os.chdir("src")
    assert handler(event, context) == "value1"

Before you run pytest for the first time in this project (or before you do any local development that needs any of the packages in the [tool.poetry.group.dev.dependencies] section of pyproject.toml) you need to run at least once:

poetry install

This will install all the dependencies listed in your main [tool.poetry.dependencies] and in [tool.poetry.group.dev.dependencies] in the virtualenv that is managed by Poetry.

You can run the test from services/my_lambda:

poetry run pytest

References on src layout

• PyPA Python Packaging User Guide src layout vs flat layout
• Pytest Good Integration Practices: Tests outside application code
• The post that pretty much started the movement: Packaging a python library by Ionel Cristian Mărieș
• Another early influential blog post Testing & Packaging by Hynek Schlawack
• Organize Python code like a PRO
• Convert a Poetry package to the src layout