Simplifying Serverless Workflows with EventBridge Pipes

Introduction
Event-driven architectures (EDAs) are essential when creating scalable, decoupled systems. By allowing services to communicate asynchronously, they reduce bottlenecks and enable flexible scaling.
AWS offers many services for building EDAs, in this post I will focus on two different combinations:

1. The traditional SNS/SQS combo
2. The newer EventBridge Pipes.

We’ll explore how to build a serverless workflow using EventBridge Pipes (integrating SQS, DynamoDB Streams, and Lambda) and compare it to SNS/SQS setups.

What are eventBridge pipes?
EventBridge Pipes is a serverless integration service that connects AWS services like (SQS, DynamoDB Streams, and Lambda) without requiring intermediate code. It supports filtering, enrichment (we use Lambda to do it), and transformations, EventBridge Pipes simplifies creating event-driven workflows.

The diagram below illustrates the workflow of AWS EventBridge Pipes, which enables seamless integration between event sources and targets with optional processing steps.

Here's a breakdown of the components and their roles:

• Event Source: This is the origin of events, such as AWS services (e.g., S3, DynamoDB, Kinesis), SaaS applications, or custom applications. Events are ingested into the pipe from here.

• Filter: Events are evaluated against predefined criteria (e.g., JSON-based rules). Only events that match the filter conditions proceed to the next step, reducing unnecessary processing.

• Enrichment: Optional step to enhance the event data. For example, you might invoke an AWS Lambda function to fetch additional information from a database or transform the event payload before sending it to the target.

• Target: The final destination where processed events are delivered. Targets can be AWS services (e.g., Lambda, SNS, SQS, EventBridge event buses), HTTP endpoints, or other resources.

How it works together:

• The event source generates events (e.g., an S3 bucket upload).
• The filter pick up relevant events.
• The enrichment step adds/modifies data.
• The target receives the refined event for further action (e.g., triggering a Lambda function to process the file).

Comparing EventBridge Pipes with SNS/SQS

Now as we know how it works let's learn by looking at a use case of E-Commerce Order Fulfillment system
In this e-commerce platform every time a customer places an order, the order details are stored in a DynamoDB table. Changes in the table (e.g., new orders) are captured by DynamoDB Streams. Instead of writing custom integration code to filter and route these events, you can use EventBridge Pipes. With Pipes, you configure the following workflow:

1. DynamoDB Streams as the Source: When a new order is added to the DynamoDB table, the change event is captured.
2. EventBridge Pipes Filtering & Routing: The Pipe ingests events from DynamoDB Streams and applies a filter to select only orders with a status of "NEW." It then routes these filtered events directly to a Lambda function.
3. Lambda Function for Order Processing: The Lambda function processes the order (for example, updating inventory, initiating shipping, and sending notifications).

This approach is compared to the traditional setup where events are published to an SNS topic and then pushed to an SQS queue before being processed by a Lambda function.

Let's view both architectures and explain the advantages of each one, and when to use each one of them:

EventBridge Pipes approach

Diagram Explanation

• Sources:
  1. DynamoDB Streams: Capture changes from a table (for example, new or updated order records).
  2. SQS Queue (if needed): Acts as an additional source that may buffer events from various systems.
• EventBridge Pipes: Serves as the central hub that filters, transforms, and routes incoming events from both DynamoDB Streams and SQS.
• Lambda Function: Consumes the refined events to process business logic (e.g., updating order status or triggering further workflows).

Traditional SNS-SQS approach

Diagram Explanation

• SNS Topic: Typically used to broadcast events to multiple subscribers.
• SQS Queue: Buffers messages and provides a pull-based delivery mechanism.
• Lambda Function: Processes the events retrieved from the SQS queue.

Comparing EventBridge Pipes with SNS/SQS

EventBridge Pipes

Advantages:

• Integrated Filtering & Transformation as you can set up rules directly within Pipes to process events without extra Lambda functions.
• Simpler Configuration because it directly connects sources like DynamoDB Streams or SQS to Lambda with minimal configuration.
• Reduced operational complexity by eliminating extra intermediary services.

When to Use:

• When your workflow is relatively linear or requires simple filtering.
• When you prefer a code-free, declarative integration between event sources and targets.
• For pipelines where transformation and routing can be managed entirely within EventBridge Pipes.

Traditional SNS/SQS

Advantages:

• Fan-Out Capabilities as SNS naturally supports broadcasting a single event to multiple subscribers.
• Services reliability with fine-grained delivery and retry control.
• Suitable for complex scenarios that require multiple consumer endpoints.

When to Use:

• When your architecture needs to fan out events to multiple consumers.
• If you require detailed control over message delivery policies and retry behaviors.
• When your existing setup is already built around SNS/SQS patterns and migrating to Pipes isn’t practical.

Cost Components of Each Approach

EventBridge Pipes Approach

AWS Service	Cost Component
DynamoDB Streams	First 2.5M reads free per month, then $0.02 per 100,000 reads
EventBridge Pipes	$0.40 per 1M events processed
Lambda (Order Processing)	$0.20 per 1M requests + execution time cost (depends on memory & duration)

SNS/SQS Approach

AWS Service	Cost Component
DynamoDB Streams	$0.02 per 100,000 reads
SNS	$0.50 per 1M publishes
SQS (FIFO Queue)	$0.50 per 1M requests
Lambda (Order Processing)	$0.20 per 1M requests + execution time cost

Conclusion
EventBridge Pipes offer a modern, efficient way to build decoupled, event-driven workflows on AWS. By integrating sources like DynamoDB Streams and SQS directly with Lambda, you can streamline your architecture and reduce operational overhead. However, traditional SNS/SQS setups remain a good choice for scenarios that require broad event broadcasting and detailed control over message delivery.

The right choice depends on your specific use case: go for EventBridge Pipes when you seek simplicity and reduced code, or stick with SNS/SQS for more complex, fan-out scenarios. Either way, understanding these patterns is key to building scalable and resilient serverless applications.