Trigger step functions from lambda after uploading files to S3 using Serverless Framework and Python

In this post, I'm gonna show you how to execute a step function from a lambda function after a file is uploaded to an S3 bucket.

You can watch the video version here:

Intro

In the image, we can see we have three AWS services, S3, Lambda Functions, and Step Functions.

After a file is uploaded to an S3 bucket, AWS will send an event to a lambda function passing the bucket and file name as part of the event.

Steps

Let's start by creating a serverless application:

serverless create --template aws-python3 --path my-state-machine

Now let's go inside the created folder my-state-machine

cd my-state-machine

And open it up in your favorite IDE. I'm using vscode
With vscode you can open your project directly from the terminal with the command:

code .

Now let's open the serverless.yml file and remove all comments for clarity.
After removing all comments we'll have a file similar to this:

service: my-state-machine

provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    handler: handler.hello

Our next step is to create and configure our Lambda function.
Let's replace the functions section with the following snippet:

functions:
  StateMachineTrigger:
    handler: handler.upload
    events:
      - s3:
          bucket: ${self:service}-${self:provider.stage}-upload-bucket
          event: s3:ObjectCreated:*
    environment:
      MY_STATE_MACHINE_ARN: ${self:resources.Outputs.MyStateMachine.Value}          
    iamRoleStatements:
      - Effect: "Allow"        
        Action:
          - states:*
        Resource: "*"

The events section is declaring that every time a file is uploaded (ObjectCreated) into our bucket ${self:service}-${self:provider.stage}-upload-bucket AWS will send a message to our lambda function lib/trigger.handler with the bucket and file information.
You can see we are declaring an environment variable MY_STATE_MACHINE_ARN. This is for our lambda function to know what is the step function's ARN.

Now we need to create the lambda handler we just declared so lets
replace the file handler.py with the following:

import os
import json
import boto3

my_state_machine_arn = os.environ['MY_STATE_MACHINE_ARN']
client = boto3.client('stepfunctions')


def upload(event, context):
    print(event)
    for record in event['Records']:
        response = client.start_execution(
            stateMachineArn=my_state_machine_arn,
            input=json.dumps(record['s3'])
        )

We are using AWS boto3 to trigger our step function that we are going to declare in just a second, and so we are creating a client for that. We are also declaring a variable my_state_machine_arn to access the environment variable we declared in our serverless.yml. And finally, we are using the start_execution method to execute our lambda function passing the event's s3 (event_record['s3']) object as argument.

Our final step is to declare the step function.
To do that we are going to use a plugin named serverless step functions so let's add the following to the end of our serverless.yml:

stepFunctions:
  stateMachines:
    MyStateMachine:
      id: MyStateMachine
      definition:
        StartAt: ValidateImage
        States:
          ValidateImage:
            Type: Pass
            Result: "Valid"
            Next: GenerateThumbnail
          GenerateThumbnail:
            Type: Pass
            Result: "Thumbnail created"
            Next: NotifyUser
          NotifyUser:
            Type: Pass
            Result: "Notification sent"
            End: true

resources:
  Outputs:
    MyStateMachine:
      Description: The ARN of the example state machine
      Value:
        Ref: MyStateMachine            


plugins:
  - serverless-step-functions
  - serverless-iam-roles-per-function

package:
  individually: true
  exclude:
    - '**/*'
  include:
    - handler.py

This is just a dummy state machine to simulate a use case when we have do the following tasks

• ValidateImage: you could validate if the image is from a celebrity, or cats, dogs, etc
• GenerateThumbnail: generate a thumbnail of the image
• NotifyUser: send an email to the user with validation feedback

The Outputs section is to get the step function's ARN we are passing in the lambda environment variable

It is recommended and a best practice to always use the plugin serverless-iam-roles-per-function for all your serverless applications

Before we deploy our application we need to install our serverless plugins and for that, we need to create a package.json file first, so let's do:

npm init -f

And now we install the plugins:

npm install serverless-step-functions
npm install serverless-iam-roles-per-function

After that, we are ready to deploy our application so let's go ahead and run:

serverless deploy

Now let's go to the AWS console, and upload a file in the bucket created and you will see that your state machine is executed

if you liked this post don't forget to click on ❤️

About me

Raisel Melian’s Twitter
Raisel Melian’s Linkedin
Raisel Melian’s YouTube channel