CiCube for CICube

Posted on Dec 11 • Originally published at cicube.io

What is AWS Step Functions? - A Complete Guide

#aws #serverless #devops #security

Introduction

Quick Summary: What You Need to Know About AWS Step Functions

What is AWS Step Functions and how does it work?

AWS Step Functions is a serverless orchestration service that simplifies workflow management by connecting AWS Lambda and other AWS services. It visually organizes tasks and automates complex processes for seamless execution.

What are common use cases for AWS Step Functions?

E-commerce workflows: Order validation, inventory checks, and payment processing.
Data processing: Managing large data sets with parallel tasks.
Error handling: Retrieving tasks and managing failures in critical processes.
Automation: Automating multi-service operations efficiently.

How do you design a workflow with AWS Step Functions?

Define your workflow: Use Amazon States Language (ASL) to structure steps.
Add state types: Include Task, Choice, or Parallel states for specific actions.
Handle errors: Use Catch and Retry blocks to manage failures.
Integrate services: Connect with AWS services such as Lambda, DynamoDB, and S3.

What are the benefits of AWS Step Functions?

Simplified workflows: Visually organize and manage tasks.
Built-in error handling: Retry and catch mechanisms for reliability.
Parallel execution: Simultaneous task execution for efficiency.
Cost efficiency: Optimize workflows to minimize resource usage.

Now that we've covered the key takeaways and provided a quick overview, let’s dive into more detailed applications and real-world scenarios. During my years of working on serverless applications, one thing that I used to feel was that managing numerous Lambda functions along with other services became complex really fast.

AWS Step Functions is one of those game-changing services that has completely changed how I approach this problem.
Today, I want to share my experience with Step Functions and how it can simplify your serverless workflows.

Steps we'll cover:

Understanding Step Functions
Key Concepts
Real-World Example: Order Processing System
AWS Step Function Best Practices
Advanced Features
Performance Optimization
Cost Optimization
State Types - My Implementation Guide

Understanding Step Functions

At its core, Step Functions is a serverless orchestration service that lets you combine AWS Lambda functions and other AWS services into business-critical applications. Think of it as a conductor in an orchestra, coordinating different services to work together harmoniously.

Here is a simple example of what a Step Function state machine looks like:

{
  "Comment": "A simple order processing workflow",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:validateOrder",
      "Next": "CheckInventory"
    },
    "CheckInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:checkInventory",
      "Next": "ProcessPayment"
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processPayment",
      "End": true
    }
  }
}

Monitoring GitHub Actions Workflows

CICube is a GitHub Actions monitoring tool that provides you with detailed insights into your workflows to further optimize your CI/CD pipeline. With CICube, you will be able to track your workflow runs, understand where the bottlenecks are, and tease out the best from your build times. Go to cicube.io now and create a free account to better optimize your GitHub Actions workflows!

Key Concepts

Let me break down the essential concepts that I work with day in and day out.

State Machines

State machines are the core of Step Functions. They define your workflow using Amazon States Language (ASL). Each state machine contains:

States: This means individual steps in your workflow.
Transitions: Rules for transitioning from one state to another Input/Output Processing: Data manipulation between states

State Types

I use these kinds of states quite often:

Task States: Execute work (Lambda, AWS services)

{
  "ProcessOrder": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processOrder",
    "Next": "SendNotification"
  }
}

Task states are used to perform specific tasks, like running a Lambda function or invoking an AWS service. In this example, the processOrder Lambda function is executed, and the workflow then moves to SendNotification.

Choice States: You can add branching logic

{
  "CheckOrderValue": {
    "Type": "Choice",
    "Choices": [
      {
        "Variable": "$.orderValue",
        "NumericGreaterThan": 100,
        "Next": "ApplyDiscount"
      }
    ],
    "Default": "ProcessNormally"
  }
}

Choice states add branching logic to your workflow. Here, the workflow checks if the orderValue is greater than 100. If true, it goes to ApplyDiscount. Otherwise, it defaults to ProcessNormally.

Parallel States: Parallel execution of the branches

{
  "ProcessOrder": {
    "Type": "Parallel",
    "Branches": [
      {
        "StartAt": "UpdateInventory",
        "States": {
          "UpdateInventory": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:updateInventory",
            "End": true
          }
        }
      },
      {
        "StartAt": "SendNotification",
        "States": {
          "SendNotification": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:sendNotification",
            "End": true
          }
        }
      }
    ],
    "End": true
  }
}

Parallel states allow multiple tasks to run simultaneously. In this example, two branches are executed at the same time: UpdateInventory and SendNotification. The workflow waits for both branches to complete before moving to CompleteOrder.

Real-World Example: Order Processing System

Let me elaborate on one recently implemented by me. This workflow handles an e-commerce order processing system:

{
  "Comment": "E-commerce Order Processing Workflow",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:validateOrder",
      "Next": "CheckInventory"
    },
    "CheckInventory": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:checkInventory",
      "Next": "ProcessPayment"
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processPayment",
      "Next": "FulfillOrder"
    },
    "FulfillOrder": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "UpdateInventory",
          "States": {
            "UpdateInventory": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:updateInventory",
              "End": true
            }
          }
        },
        {
          "StartAt": "SendConfirmation",
          "States": {
            "SendConfirmation": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:sendConfirmation",
              "End": true
            }
          }
        }
      ],
      "End": true
    },
    "HandleError": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:handleError",
      "End": true
    }
  }
}

This workflow encompasses:

Error handling with Catch blocks
Retry logic for transient failures
Independent tasks executed in parallel - State transitions based on business logic

AWS Step Functions Best Practices

From this experience, I have developed the following best practices:

Error Handling

Always retry on transient failures
Employ Catch blocks to handle errors gracefully
Log state transitions in debug

State Machine Design

Keep state machines focused and single-purpose
Make use of built-in error handling in Step Functions instead of implementing error handling in Lambda
Leverage parallel states for independent operations

Input/Output Processing

InputPath and OutputPath to filter data
Implement ResultSelector to shape task output - Keep payload size below 256 KB

Monitoring and Debugging

Enable CloudWatch detailed logging - Use X-Ray for tracking - Configure CloudWatch alarms on failed executions

Advanced Features of AWS Step Functions

Some of the frequently used advanced features:

Dynamic Parallelism

Dynamic Parallelism lets you process multiple tasks at once, even if you don’t know how many tasks there will be ahead of time. It's perfect for handling scenarios like processing a list of items that keeps changing.

Using the Map state, you can run tasks in parallel.

{
  "ProcessBatch": {
    "Type": "Map",
    "ItemsPath": "$.items",
    "MaxConcurrency": 10,
    "Iterator": {
      "StartAt": "ProcessItem",
      "States": {
        "ProcessItem": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processItem",
          "End": true
        }
      }
    },
    "End": true
  }
}

Breaking it down:

Map state: Handles parallel processing for a list of items.
ItemsPath: Points to the array of items in your input JSON.
MaxConcurrency: Sets how many tasks can run at the same time.
Iterator: Defines the steps to follow for each item in the list.

Why it’s awesome:

This setup is great for tasks like resizing images, processing payments, or transforming data. It’s make sure our system stays efficient by running tasks in parallel without overloading it.

Integration Patterns

{
  "WaitForCallback": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
    "Parameters": {
      "FunctionName": "arn:aws:lambda:REGION:ACCOUNT:function:longRunningTask",
      "Payload": {
        "taskToken.$": "$$.Task.Token"
      }
    },
    "Next": "ProcessResult"
  }
}

This example shows how we can pause a workflow until a task finishes and sends a response. It’s perfect for long-running tasks where you need to wait for a callback before moving forward.

Type: Task: Defines this as a task state.
Resource: Uses a special ARN to call a Lambda function and wait for a callback with a task token.
Parameters:
- FunctionName: Points to the Lambda function handling the long-running task.
- taskToken.$: A unique token automatically generated by AWS Step Functions for this task. It’s included in the payload sent to the Lambda function.

How it works:

When the workflow reaches this task state, it invokes the Lambda function.
The Lambda function receives the taskToken in the payload.
The Step Function pauses and waits for the Lambda function to send a callback with the token.
Once the callback is received, the workflow resumes and moves to the ProcessResult state (defined in the Next field).

Why we use this setup?

It's ideal for scenarios like manual approvals or asynchronous tasks where the next step depends on external input or a long-running process. The workflow remains efficient by pausing instead of polling or retrying.

Performance Optimization

Based on my experience with Step Functions performance optimization, here are some best practices that I have learned:

Optimize State Transitions

I find that the transitions among states have a significant impact on cost and performance; the following is how I optimize them:

{
  "ProcessOrder": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processOrder",
    "ResultsSelector": {
      "relevantData.$": "$.specificField"
    },
    "Next": "NextState"
  }
}

I used ResultsSelector to map only the needed data between states
I keep my payload size less than 256KB across states
I combine states when possible to reduce transitions

Lambda Optimization

Since I develop with Lambda functions pretty often, here go some few optimization tricks:

I make adjustments to Lambda memory based on the workload I am processing
I use Provisioned Concurrency for frequently used Lambdas
I used the timeouts for each state according to my experience.

Parallel Processing Strategies

This pattern I use when I have multiple items to process:

{
  "ProcessBatch": {
    "Type": "Map",
    "MaxConcurrency": 10,
    "ItemsPath": "$.items",
    "Iterator": {
      "StartAt": "ProcessItem",
      "States": {
        "ProcessItem": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processItem",
          "End": true
        }
      }
    }
  }
}

I set optimal MaxConcurrency based on my workload - I batch small operations together - I use DynamoDB batch operations for volume activities

Cost Optimization for AWS Step Functions

Here is how I keep my Step Functions costs under control:

State Transition Costs

I've learned that each state transition costs something:

Standard Workflow: $0.025 per 1,000 state transitions Express Workflow: Charged depending on usage duration and memory

Here's what I do to optimize costs:

Select Workflow Type:

I use Standard Workflow for long-running, low-state processes
I use Express Workflow for most of my high-volume, short-duration tasks.

State Combination:

{
  "CombinedProcessing": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:combinedProcessor",
    "Next": "FinalState"
  }
}

Whenever possible, I combine small states.

Lambda Cost Optimization

I balance memory and duration for optimal cost
Use batch processing to reduce the number of Lambda invocations

Service Integrations I use integrations of direct service to reduce costs:

{
  "WriteToDynamoDB": {
    "Type": "Task",
    "Resource": "arn:aws:states:::dynamodb:putItem",
    "Parameters": {
      "TableName": "MyTable",
      "Item": {
        "id": {"S.$": "$.id"},
        "data": {"S.$": "$.payload"}
      }
    },
    "Next": "NextState"
  }
}

This codeblock shows how to write directly to a DynamoDB table using AWS Step Functions without requiring a Lambda function.

Type: Task: Specifies this as a task state in the workflow.
Resource: Connects directly to DynamoDB’s putItem operation.
Parameters: - TableName: The name of the DynamoDB table where data will be written. - Item: Maps the input values (e.g., id and payload) to the corresponding columns in the table.

Ok but how it works?

The workflow writes the specified data directly to DynamoDB when it reaches this task state. Once the putItem operation completes, the workflow transitions to the next step, as defined in the NextState.

Needless to say, doing away with Lambda for workflows that would require having to use Lambda can be simplified by directly integrating AWS Step Functions with DynamoDB. In this case, the it will now directly interact with the services over AWS. Therefore, transactions are quicker and much less costly-the best possible case for workflows, which are meant only for data storage in DynamoDB.

State Types - My Implementation Guide

How I use state types differently in my workflows:

Task States

I use Task states to do the actual work:

{
  "ProcessPayment": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:processPayment",
    "TimeoutSeconds": 30,
    "Retry": [
      {
        "ErrorEquals": ["ServiceException"],
        "IntervalSeconds": 2,
        "MaxAttempts": 3,
        "BackoffRate": 1.5
      }
    ],
    "Catch": [
      {
        "ErrorEquals": ["States.Timeout"],
        "Next": "HandleTimeout"
      }
    ],
    "Next": "NextState"
  }
}

Key features that I always set up:

I set timeouts based on the expected duration
I add retry logic for transient failures
I handle errors with Catch
I do the filtering of output data with ResultsSelector

Choice States

I use Choice states to make a decision:

{
  "EvaluateOrder": {
    "Type": "Choice",
    "Choices": [
      {
        "And": [
          {
            "Variable": "$.orderValue",
            "NumericGreaterThan": 1000
          },
          {
            "Variable": "$.customerType",
            "StringEquals": "premium"
          }
        ],
        "Next": "ApplyPremiumProcess"
      },
      {
        "Variable": "$.orderValue",
        "NumericLessThan": 100,
        "Next": "ApplyFastProcess"
      }
    ],
    "Default": "StandardProcess"
  }
}

I use them for:

Routing based on business logic
Data validation
Conditional processing

Parallel States

When I need to execute several tasks simultaneously that are independent of each other:

{
  "ProcessOrder": {
    "Type": "Parallel",
    "Branches": [
      {
        "StartAt": "UpdateInventory",
        "States": {
          "UpdateInventory": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:updateInventory",
            "End": true
          }
        }
      },
      {
        "StartAt": "NotifyCustomer",
        "States": {
          "NotifyCustomer": {
            "Type": "Task",
            "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:notifyCustomer",
            "End": true
          }
        }
      }
    ],
    "Next": "CompleteOrder"
  }
}

Important things I have learned: Each branch runs independently - Next state waits for all branches - Each branch needs its own error handling

Conclusion

AWS Step Functions changed how I build serverless applications. It gave me a robust way to orchestrate even the most complex workflows-keep clear, maintainable configurations. The visual workflow editor, combined with the power of Amazon States Language, makes it so much easier to design, implement, and maintain serverless applications.

Remember, the key to successful Step Functions implementation:

Clear workflow design
Proper error handling
Efficient state management Comprehensive monitoring

If you are building serverless applications on AWS, I highly recommend checking out Step Functions; it may turn out to be the missing piece in your serverless architecture puzzle.

DEV Community