After working extensively with microservices, I’ve come to appreciate the power and flexibility of the Event-Driven Architecture (EDA). EDA is a software design pattern that allows systems to detect, process, and react to real-time events in a decoupled manner. It is particularly suited for microservices, enabling seamless communication and interaction between independent components.
In this article, we’ll explore the principles of event-driven architecture, discuss its core concepts, and demonstrate how it works using a custom-built command-line tool. This tool, written in Go and powered by Redis, serves as a practical example of EDA in action.
To follow along with the demonstrations, you’ll need the following:
- Download the CLI tool from GitHub (available for Windows, Linux, and macOS) Make sure to download the latest release.
- Have a Redis server running locally. You can install Redis from the official website or use Docker for easier setup (recommended):
How the Event-Driven Architecture Works
At a high level point of view the EDA is quite straight forward. Events representing occurrences or changes in the system drive the flow are generated by various sources (Publishers), published to an event bus or message broker, and consumed by interested components (Consumers) asynchronously. This approach promotes flexibility, scalability, and resilience.
An event is any change in state of the system and can be anything ranging from customer requests, inventory updates, sensor readings, and so on.
The event-driven architecture enables different components of a system to run independently of each other by introducing a middleman known as an Event broker that routes events to different intended destination components. This means that applications and devices do not need to know where they are sending information or where the information they are consuming comes from. Rather they focus on carrying out there intended functionality.
Note: In this article we use Redis because of its simplicity and realatively easy learning curve but Redis wont demonstrate the full functionality of an Event Driven System, however there are other more complex software systems such as RabbitMQ or Apache Kafka that provide a wide range of functionality and Protocals such as AMQP and MQTT(Commonly used in IoT application) that can be used to build more large scale systems, however these have a steep learning curve and wont be ideal for demonstration purposes in this article.
Concepts
With this high level understanding of how the Event-Driven Architecture works, lets look and some of its core concepts. These will give you a platform to weigh the strengths of this architecture to others and possibly give you a basis on deciding when to apply it in your projects. We sha;; use the tool to demonstrate these concepts.
The Event broker
An Event Broker is a middleware platform that routes events from source to desired destination. This is accomplished using the publish-subscribe messaging pattern. In this pattern, independent system components connect to the event broker and exchange messages through the Event Broker.
Multiple event brokers Exist, some of the most popular ones include Apache Kafka, RabbitMQ, AWS EventBridge, and Redis. In this article, we will use Redis as our event broker. To follow along with the practical example atart you redis instance. Make sure its listening on port 6379 as thats where eda cli tool will try to access it from. Unfortunately I had to dard code the port value. You can run the following command to run Redis in a Docker Container.
docker run -d -p 6379:6379 redis
Event Portals
Event portals in Event-Driven Architecture are tools or platforms designed to facilitate the management, discovery, and governance of events within an event-driven system. They are especially valuable in modern distributed systems where events act as the primary means of communication between components.
You should not confuse them with Event Brokers as the definition might sound similar. Here are the differences
| Aspect | Event Portal | Event Broker |
|---|---|---|
| Primary Function | Discovery, governance, and design of events | Real-time routing and delivery of events |
| Focus | Design-time and metadata management | Runtime event processing |
| Scope | Entire event-driven system | Specific event transportation tasks |
| Examples | AsyncAPI, Solace Event Portal | Kafka, RabbitMQ, AWS EventBridge |
I found a good article on the Internet explaining Event Portals in detail. Read it here
Event Mesh
An event mesh is created and enabled through a network of interconnected event brokers. It’s a configurable and dynamic infrastructure layer for distributing events among decoupled applications, cloud services, and devices by dynamically routing events to any application, no matter where these applications are deployed in the world, in any cloud, on-premises, or IoT environment. Technically speaking, an event mesh is a network of interconnected event brokers that share consumer topic subscription information and route messages amongst themselves so they can be passed along to subscribers.
Topics in Event-Driven Architecture
In an event-driven system, topics are labels used to organize and route events. When an event is published, it is associated with a specific topic, and consumers subscribe to topics to receive events relevant to them. This approach allows producers to send events without needing to know which consumers will process them, while consumers only handle events tied to their subscribed topics. For example, an event with the topic orderCreated could be published by a producer, and any consumers subscribed to orderCreated would receive and process the event.
In this demonstration, we will set up a system with two components: the API server, which acts as the producer, and the consumer, which listens to a topic and logs the events it receives. First, you’ll need to start the API server by running the following command in your terminal:
eda api-server
This command initializes the server, which listens for POST requests. These requests include events that the server will publish to Redis, using the specified topic. Next, you’ll start a consumer that subscribes to a particular topic. To subscribe to the topic orderCreated, use the following command:
eda consumer --topic orderCreated
With both the API server and the consumer running, you can now send an event to the system. Use an HTTP client, such as curl or Postman, to send a POST request to the API server. For instance, sending the following request publishes an event to the topic orderCreated:
curl -X POST -H "Content-Type: application/json" \
-d '{"order_id": 123, "customer": "John Doe", "total": 49.99}' \
http://localhost:3000/topics/orderCreated
When this request is received, the API server sends the event to Redis, the event broker. Redis routes the event to all consumers subscribed to the topic orderCreated. If a consumer is actively listening, it will receive the event and log the message to its console. The consumer’s output will look like this:
Received event on topic: orderCreated
Event data: {"order_id": 123, "customer": "John Doe", "total": 49.99}
This demonstration shows how topics allow events to be routed to the appropriate consumers without requiring direct communication between the producer and the consumers. By associating events with topics, the system achieves flexibility and scalability while keeping its components decoupled.
Deferred Execution
Deferred execution is a key concept in event-driven architecture that differs significantly from traditional REST-based APIs. In a REST-based system, when you send a request, you typically wait for an immediate response. However, in event-driven systems, you don’t wait for a response when you publish an event. Instead, the event is handed off to an event broker, which holds or persists the event until consumers are ready to process it. This allows for more flexible and decoupled communication, as consumers can process events at their own pace, potentially much later.
In this approach, acting on one event might also cause new events to be generated and persisted in the event broker. These subsequent events can trigger further actions in other parts of the system, creating a chain of reactions, all while maintaining the flexibility of deferred processing.
To understand this concept better, let’s walk through a practical demonstration of deferred execution.
Practical Demonstration
In this demonstration, you will observe how events can be held in a queue and processed after a delay, demonstrating deferred execution. Here are the steps:
- Start the API Server
First, start the API server, which allows you to publish events to the system:
eda api-server
This server listens for POST requests and routes event messages to the system.
- Send an Event
Send a POST request to the /deferred-exec endpoint of the API server. The request body contains the event message you want to publish. For example:
curl -X POST -H "Content-Type: application/json" \
-d '{"task_id": 456, "description": "Process data files"}' \
http://localhost:3000/deferred-exec
When this request is sent, the API server pushes the event to Redis, the event broker, where it will be stored in a queue for processing.
- Start the Master Node
Next, start the master node, which plays a key role in deferred execution by sending events from Redis to the event queue. To start the master node, run the following command:
eda consumer -d --master
- Start a Consumer
Finally, run the consumer to process events from the event queue. The consumer checks the queue every 10 seconds, retrieves the events, and logs them to demonstrate deferred execution. To start the consumer, use the command:
eda consumer -d
Once the consumer is running, you will see a 10-second delay before the consumer processes events. For example, the consumer’s log might display:
Processing event: {"task_id": 456, "description": "Process data files"}
This delay illustrates how events can be stored in a queue and processed later, at the consumer’s convenience. Such deferred execution is particularly useful in scenarios where immediate processing isn’t required, allowing for greater flexibility and resource efficiency.
Eventual Consistency
Building on the concept of deferred execution, where events are processed at a later time rather than immediately, we encounter the idea of eventual consistency. In an event-driven system, since events are not processed instantly, there’s no guarantee that all components, such as databases or systems, are synchronized at any given moment. For example, if you have multiple systems—like a database, a master data management (MDM) system, and an ERP—they may not have the same state at the same time.
In our deferred execution demonstration, when an event (like a task or an order) is published to Redis, it may take time for the consumer to process it. Until the consumer acts on the event and updates the relevant systems, those systems might not reflect the latest state. However, while consistency isn’t immediate, we can rely on the system to eventually synchronize. This means that all stateful components will reach a consistent state over time, provided the system is functioning as intended and events are processed correctly.
For example, after publishing a task event with the API server, the consumer may process it after a delay, updating the system state accordingly. While the task's status may appear outdated immediately after publication, you can trust that the system will catch up and reflect the changes eventually.
This principle is particularly useful in distributed systems, where maintaining immediate consistency across all components can be impractical or inefficient. By embracing eventual consistency, we trade immediate synchronization for better scalability and resilience, trusting the system to converge to a consistent state in due time.
Choreography
In the world of event-driven architecture, choreography is a pattern where multiple components interact and coordinate their actions by responding to shared events. Unlike orchestration, where a central controller dictates the flow of operations, choreography allows each component to act independently based on the events it receives. This decentralized approach aligns well with the principles of EDA, promoting loose coupling and scalability.
Benefits of Choreography
Choreography ensures that components operate independently, reducing the risk of bottlenecks or single points of failure. In our system, the master node facilitates event flow but does not tightly couple itself to the consumers. This allows for easy scaling—new consumers can join without the master node needing to know about them, as long as they subscribe to the appropriate topics or queues.
By adopting choreography, you can build systems that are more resilient and flexible. Each component focuses on its responsibilities, reacting to events as needed, rather than relying on a central controller to orchestrate every action.
CQRS:Command Query Responsibility Segregation
CQRS is a method for scaling microservices by separating responsibilities. One service handles commands, such as updates or inserts, while another handles queries, such as fetching data. This separation is useful because query services often need to handle far more requests than commands, making it easier to scale the query service independently.
In an event-driven architecture, implementing CQRS is straightforward. Events in the system can be categorized by topics that include a specific action or verb, like "query" or "update." To scale the query service, you can simply deploy more instances of it and have them listen to the topics related to queries. This way, the system remains efficient, scalable, and easy to manage.
Conclusion
Event-Driven Architecture (EDA) is a powerful paradigm that enables systems to be more flexible, scalable, and resilient by decoupling components and allowing them to communicate through events. Throughout this article, we’ve explored the principles of EDA, discussed its core concepts like deferred execution, eventual consistency, choreography, and CQRS, and demonstrated how these concepts can be applied in practice using Redis and a custom-built CLI tool.
By embracing EDA, you can design systems that not only handle complex workflows efficiently but also adapt to changing demands with ease. Whether you're scaling query services, coordinating actions through events, or ensuring eventual consistency in distributed systems, EDA provides the tools and patterns to build robust architectures.
Thank you for taking the time to read this article! I hope it has been informative and inspires you to experiment with EDA in your own projects. About me, I am Beingana Jim Junior, a software engineering student at Makerere University, I love sharing insights from my learning and experiences. I’m passionate about software design and always eager to explore new technologies and solutions. Check out my Github and Portifolio website
Feel free to share your thoughts, questions, or feedback. I’d love to hear from you. Let’s continue building amazing systems together!
References
Event-Driven Architecture. Amazon Web Services. https://aws.amazon.com/event-driven-architecture/
Event-Driven Architecture (EDA) - A Complete Introduction. Confluent. https://www.confluent.io/learn/event-driven-architecture/
The Complete Guide to Event-Driven Architecture. Medium. https://medium.com/@seetharamugn/the-complete-guide-to-event-driven-architecture-b25226594227
Top comments (0)