Getting Started with AWS Lambda Layers

Once you begin working with Lambda functions there will inevitably come a time when you have to include an external library in your function. Out of the box Lambda functions only support a limited number of libraries with their runtimes. In order for your function to access those external dependencies you would have to include them in your deployment package. Each time you deploy a Lambda function making sure to include external dependencies in the deployment package was a part of the workflow. During this step when deploying a Lambda function I'd catch myself thinking, "there has to be a better way". And now there is! With the introduction of layers for AWS Lambda you no longer have to bundle your external dependencies with your deployment package. Layers allow you to manage and share your external dependencies across your Lambda functions. In this post I'm going to,

1. Explain what layers are and their benefits
2. Walkthrough how to create and use a layer in a Lambda function
3. Provide some best practices and tips

Even though this post will largely be focused on Python, a lot of the core lessons are extensible to other languages supported by AWS as well. Before getting started a quick disclaimer. AWS Lambda functions have a bunch of built-in nuances and limitations. While layers are helpful they are not a panacea. Please take a minute to read the last section about best practices. As with any tool it becomes useless if you're not using it properly, so taking a minute or two to understand the limitations can save you from a headache and frustration later on.

Introducing AWS Layers

So, what are these layers and how are they useful? An AWS Lambda layer contains additional code, libraries and dependencies that are loaded to support a Lambda function at runtime. Layers eliminate the need to package external dependencies in your deployment package for each Lambda function you create. A layer is a package of all the external dependencies your Lambda function requires. At runtime the dependencies specified in the layer are loaded and available to your Lambda function. Layers are not specific to individual Lambda functions either. Once you create a layer you can use that layer to support multiple different Lambda functions. There is no need to recreate a layer for each Lambda function. Each Lambda function can support up to 5 different layers providing the flexibility to mix-and-match different layers.

Layers also support versioning making them easier to manage. When collaborating with others it is easier to have a centralized location for Lambda dependencies that can be referenced instead of each person building their own dependencies at deployment where there is a risk package versions could become out-of-sync and potentially incompatible. With a little background of what layers are let's create one of our own!

Creating an AWS Layer

Now that you have an idea of what a layer is and why you might want to use it, let's create one. For this initial example we're going to create a layer that makes the flask framework available to our Lambda function. To build a layer there are three general steps,

1. Build the zip file with all the dependencies for the layer
2. Upload the zip file to the layer in AWS
3. Associate a layer with a Lambda function

Let's get started with creating the deployment package for our layer.

Building the Code Package

The deployment package for our layer is a zip file containing all of the code we want to include in the layer. The process is very similar to how you would incorporate dependencies into your Lambda deployment package, but the difference is with layers you do this process once instead of each time you deploy. The code here could be custom code or any other external Python packages themselves (note I'm using Python here, but layers support all different kinds of runtimes). In this introduction we're going to keep our deployment simple and will be creating a layer that makes the flask library available to our Lambda. What we need to do is install flask into a directory and then create a zip archive of that directory. One requirement when building the code package for a layer is that our code must be nested within a folder supported by the runtime for your Lambda. In our case we'll be using Python, so the flask code must be in a zip file nested within a sub-directory named python. We can do all of this from the terminal.

First, in the terminal let's create a directory for our project, which we'll call flask-layer with a python folder as a sub-directory.

mkdir -p flask-layer/python

Next, install the flask package into the flask-layer/python directory,

pip3 install flask -t flask-layer/python

Finally, zip the contents of flask-layer into a zip archive.

cd flask-layer
zip -r aws-flask-layer.zip python

Creating our Layer in AWS

Next, we can upload our code to AWS. Within the AWS console go to the Lambda service page. On the left menu select "Layers".

On the next page click "Create Layer" in the upper right corner. Now we'll define our layer. I named the layer flask-layer with a brief description. The next piece of information we need to enter is the "Code entry type". Notice the warning beneath the dropdown. For this example our code package is much less than the 10MB limit so we're going to choose the "Upload a .zip file" option then click the "Upload" button and select the aws-flask-layer.zip file you created above. If your .zip file is > 10MB you will need to upload the file to S3 and then provide the path to the file in S3. Finally, we need to associate our layer with a runtime. We'll use the Python 3.7 runtime in this example, but you can associate a single layer with up to 5 different runtimes. When you're done your page should look something like this,

At the bottom right click "Create Layer". Now that we've created the layer let's put it to use.

Using our Layer in a Lambda Function

Go back to the Lambda dashboard and click "Create function" in the upper right. Our function is going to be named, testFlaskLayer. We're going to use the Python 3.7 runtime to match the runtime we selected for our layer. If you select a different runtime than one you selected for your layer then you will not be able to access the layer from your function. Pick an existing role if you have one that has Lambda permissions, or allow AWS to create one for you and click "Create function" at the bottom.

In the next page we need to connect the layer we created with our Lambda function. Below the box that says "testFlaskLayer" in the middle of the page click the "Layers" box underneath.

Scroll down the page and click the "Add a layer" button.

From the dropdowns select flask-layer and then specify we're using version 1 of the layer, then click "Add".

Back at the Lambda designer click "Save". Now our Lambda function has access to the flask package! To make sure this is the case let's do a quick test.

In the designer scroll down to the in-line editor and paste in the following code. All we're doing is demonstrating that we're able to load the flask library because of the layer we added and we're going to just output the version of flask we're using.

import json
import flask

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': json.dumps(flask.__version__)
    }

Save the Lambda function. Now scroll to the top and run the "Hello World" test. You should get a successful result with the version of flask you're using in the body of the output. If you were to go back into your layers and remove flask-layer from your Lambda, save the function, and rerun the test you'll get a failure. By removing the layer the Lambda function no longer can access the flask library. When Lambda tries to import the flask library in this case an error will be thrown.

If you had other Lambda functions that needed to access flask all you would need to do is add flask-layer to the Lambda function and you'd be all set!

Best Practices and Other Tips

To get the most out of layers keep the code making up any single layer narrowly defined. Having narrowly defined layers limits your exposure of including unused dependencies when you start sharing layers across functions. Dependency bloat within a Lambda has real costs in terms of added execution cost, time, and performance. Narrowly defining your layers can help you avoid those costs. Another reason you'll want to smaller layers is because layers are not a work around to the Lambda deployment package size. I'll admit when I started using layers there was a fleeting moment where I thought layers could get around those limits, but that is not the case. Layers included in a Lambda will count towards the deployment size of the Lambda function contributing towards the hard limits for deployment package size of Lambda functions. Because of these size restrictions keeping your layers small allows you to make the most from your deployment packages by not wasting room on dependencies you don't need.

To make sure your Lambda function has all the dependencies that are needed stack your layers. Lambda functions allow you to include up to 5 unique layers. Stacking layers lets you mix and match the individual layers you have created to create a custom set of dependencies for your function.

Finally, one other note about creating layers to be aware of. Any code you include in your layer zip file must be compatible with AWS. As described in this issue some python packages (i.e. numpy) contain compiled code that is incompatible with AWS. The workaround for these instances is to build your deployment package with a precompiled .whl files instead of using pip install. In the case of numpy and scipy Amazon has created a public layer anyone you can incorporate into your code.

Wrapping Up

Layers are a tool to help manage your external dependencies for your AWS Lambda functions. They provide a systematic solution to sharing dependencies across Lambda functions removing the need to package those dependencies with each deployment so you can focus on the code for the Lambda function itself. Layers do have their limitations, so here are a few takeaways,

• Layers manage external dependencies for Lambda functions
• Layers can be shared across Lambda functions
• Keep layers small and leverage the ability to stack layers in functions. Smaller layers make it easier to make sure a function has only the dependencies required to perform its tasks.
• Layers are still subject to Lambda limits on code deployment size. They are not a workaround for those hard limitations.
• Dependencies must be compatible with the AWS environment, which means you may not just be able to pip install certain packages.

I hope you found this introduction to layers useful! I'm just getting started doing this, so if something isn't clear or you have further questions please let me know!

Comments

[Wolf Developer]

Interesting article! I'm completely new ti lambda layers so my doubt could be easy ti explain, but I didn't understand why layers Will count during deployment of the lambda function. I thought that the association between lambda and a layer Is like a "simple reference", so in my thought After deploying a layer of for example 20MB of dependencies, when I create a new lambda version with a code of 500KB I would deploy only that code. Is this not the case?

One good takeaway i think Is that, in the previous example, with layers I can see the lambda code in the code section, while normally I can't. Is It correct?

Anyway, thank you for your sharing, I'll investigate the subject more since seems very useful!