Written by Ably’s CEO and Co-Founder Matthew O’Riordan and published at the Ably Blog
APIs have grown to dominate the digital landscape. From the humble, RESTful beginnings of requests and responses over HTTP, we are entering an eventful period in history where end-users and business leaders (many of them digital natives) increasingly expect digital experiences to be interactive, immersive, responsive, and immediate. They insist on realtime reports of changes in state of whatever they're invested in so as to be able to make executive decisions in the moment. All this by indicating interest only once --- "subscribing" --- then having status reports land on their virtual, increasingly mobile "desk" automatically from that point onward.
The answer to these consumer demands is to design APIs using event-driven architecture (EDA) in tandem with service-oriented architectures in order to take advantage of both. In EDA, clients can subscribe to and receive events asynchronously, yet in realtime if both are online. The client no longer has to keep asking for updates or state changes.
By shifting the burden of the architectural complexity from client to producer, enterprises can more ably drive the fluid experiences that modern users have grown to expect (and demand). Event-driven architecture also results in the ability to make realtime business decisions based on the most up-to-date, bleeding-edge information.
Event-driven architecture is a recurring theme in the trends of the 20s identified by Gartner. The aughts were the decade of the web; the 10s of mobile; and this is the decade of multiexperience, including conversational, immersive, and sensory user interactions via event-driven, serverless, and edge computing architectures. Here are some of the Gartner forecasts:
- Event notifications will form part of over 60% of new digital business solutions.
- Over 50% of business organizations will participate in event-driven digital business ecosystems.
- 50% of organizations managing APIs will incorporate mediation of event notifications into their operations.
- Most leading providers of application platforms will include high-productivity tools for event-driven design.
Evolution of event-driven design
To understand event-driven APIs, we need to look at the evolution of distributed software systems architecture.
Software as remotely executing subroutines
In the early days, software programs were a subroutine with a sequence of instructions that performed a specific task. For example, ENIAC ran a subroutine that calculated the trajectory of a missile in 15 seconds.
Then came distributed computing architecture, designed around an orchestrator system that controlled the execution of multiple subroutines via remote procedure calls. Proprietary frameworks and protocols evolved to make the distributed systems work reliably.
The request-response paradigm evolved from the concept of remote procedure calls on distributed computers. If you think of software simply as several independently executing subroutines, then the request-response pattern of communication fits the application very well. An orchestrator node invokes the subroutine on a different computer with a request. The subroutine processes the request and sends back a response back to the orchestrator.
With the advent of the World Wide Web (WWW), HTTP became the go-to mechanism used to deploy distributed systems, with subroutines exposed and accessed via URLs, giving birth to the concept of APIs.
Software as a tool for real-world operations
How do you extend APIs to build software?
Let's say you have a food ordering service that manages the operational flow of a takeaway order for a restaurant. If you want to leverage the request-response paradigm of invoking APIs, then you define APIs to perform all the typical actions performed by and requested of the service: accepting orders, running inventory and reporting on it, and updating delivery status according to the latest available information.
To provision the system, you provide a web- or app-based interface. This interface would mimic the role of a human operator interacting with requestors of actions and information, be it external (customers) or internal (front-line/operations/management).
A request-response paradigm works well in this case. Instead of making a phone call, you let the users trigger the APIs through a UI to either place orders, check availability, or check the status of delivery.
Software that models real-world operations
You can further extend the food ordering service as a SaaS application that models the entire service process involved in order management. For example, you can define a set of REST APIs to create orders, tracking the order workflow from initial order till delivery. You can also manage the operations and logistics in fulfilling the order, and take care of each business transaction. In the course of handling an order, multiple objects are generated to represent the order, payment transactions, and the delivery note. These are virtual objects that model the real-world objects involved in processing an order.
With REST APIs modeling transactional objects, API design becomes more logical and nuanced, since you can query the objects directly using their specific endpoints instead of overloading a common, single endpoint with all queries. You can thus query the status of an order by querying the order endpoint, and the API returns the current state of the order. Other objects, such as the payment transaction and delivery note, are also queried through their respective API endpoints.
This kind of API design is the natural consequence of REST architecture, which defines a set of high-level guidelines to make APIs more natural to use.
But what about the communication paradigm? REST APIs also rely on the request-response pattern via HTTP. Traditionally, the onus has been on the client to request the objects for state changes as well as any other updates and notifications, continually if need be.
However, some updates, such as payment transaction failures or delays in delivery happen to be critical to the customer experience. As such, they become business-critical events. In such circumstances, it is far preferable to notify the customer about said events instead of the classic anti-pattern of customers being kept in the dark, forced to figure out what's going on all by themselves.
Sequence of supply-chain events
Very few customers appreciate having to form mental models of opaque systems; to most, it's a dealbreaker. Modern, people-centric SaaS applications need to provide higher-quality and more instantaneous interactions than those provided solely by the traditional REST approach of stateless request-response invocation.
Event-driven APIs answer the call by answering the question:
"What's the next state?"
Event-driven APIs are different from REST APIs
In the REST framework, the API does not know the state of the objects. It only serves to define the format of state representation of an object which is maintained by the API backend. The client application is responsible for maintaining the state of objects that it wants to track and is supposed to query the state by frequently polling the API. In turn, the API is only responsible for transferring the representational state of the object.
However, with an event-driven API, the client registers its intent to monitor the state with the API. The API, in turn, accepts the request and starts monitoring the state change at the backend. After that, it is supposed to initiate an event for the client whenever there is a change in state. In this case, the API backend monitors the state of objects. Moreover, it also defines the underlying transport mechanism for communicating the event to the clients that registered for the state change event.
The inversed complexity of REST and event-driven APIs
The behavior of the API thus shifts from stateless handling of repeatable, independent requests to a stateful awareness about the virtual objects that model real-world applications, operations, and entities.
Like optimizing the user's experience with a service by remembering a retail customer's name, stateful, responsive approaches to architecting services naturally complement the demands of modern mobile users.
This is easy enough to implement in one-off bespoke environments. However, offering this same level of service at scale is remarkably difficult to engineer from the ground up, and not every enterprise is ready or able to handle the associated complexities.
Scaling events and interactions is complicated.
Pros and cons of event-driven APIs
Event-driven APIs are a powerful abstraction and are applicable in a broad range of business use cases. They work well alongside traditional architectures, and come with their own pros and cons.
Pros
Modern digital interactions demand the following:
Better realtime user experiences\
Event-driven APIs lead to a better interactive experience for the end-user with modern interactivity demands by abstracting away many of the responsibilities previously assigned to users. Though this increases complexity on the producer end, the end result is worth it.
Loose coupling between API producers and consumers\
The apps consuming the API only need to know about the data format of events. Apps don't have to worry about which server is sending the data. On their side, servers are not interested in the identity of the consumers, and merely follow their internal logic to generate events. The decoupling means lower change management overhead.
Superior handling at scale\
API providers don't have to keep track of competing consumers or wait for replies from them. The number of subscriptions on the consumer end is of no consequence to the producer, who only needs to know that what they delivered to the broker is guaranteed to be passed on to the subscribers, whoever and however numerous they might be. Asynchronous and non-blocking are system characteristics ideally suited for scale, another downstream benefit of loose coupling.
Extensibility and resilience\
Loosely coupled components are also by nature easily extended in place and with low regression risks. Functionality can be added as just another independent consumer, having no impact on any other consumers in any way. Similarly, fault tolerance can be baked in from the start by design, because failure of components can be compartmentalized.
More consistent handling of concurrency\
Immutability of events and message ordering make it easier to arrive at concurrency within distributed systems.
Optimized network resource usage\
Synthesizing event-driven APIs with traditional service-oriented approaches often results in awkward constructs that tax available resources such as continuous polling. Event-driven APIs use computational and network resources more optimally by design.
Mature tech\
Event-driven approaches have been around for almost half a century and are tried and true.
Meanwhile, modern online service delivery systems have finally arrived at a point in time when they can take full advantage of event-driven architectures to meet consumer demands of realtime interaction.
Cons
Event-driven APIs still have to answer the same questions of delivering value to the end user and to business. As a complement to --- not a replacement of --- traditional SOA approaches, they bring additional burdens of domain knowledge to those who invest in and implement them. Understanding both SOA and event-driven architecture is key to a successful digital transformation.
Increased complexity for API provider\
With event-driven APIs, the API backend does all the work, not the least of which is to maintain state and keep meticulous track of all state changes. The producer takes on the entirety of the burden of orchestrating the process of delivering the end-user experience. Once a delivery is promised, events have to be published with a large degree of reliability and integrity. Implementing the system such that failures are kept statistically independent is a hard engineering problem.
Ambiguity In API analytics\
REST API calls can be metered based on the number of request hits to the API endpoint. This metric does not easily map to event-driven architecture (such as knowing the number of events being pushed, for example). Traditional ways of measuring API effectiveness and stickiness are no longer applicable in the case of event-driven APIs.
Limited governance, standardization, and developer experience/support\
API gateways currently offer no support for event-driven protocols and patterns. There is a wild variety of implementations and solutions, impeding standardization efforts. And there's a dearth of documentation/literature on how event-driven architecture can be leveraged to address exploding applications of the paradigm.
Event-driven architecture as competitive advantage
Event-driven architecture is not a new concept. It's been around since the 1970s, has been extensively battle-tested since, and has been patiently waiting for its moment. What makes this moment right is that the audience is ready, and happens to demand precisely what EDA can bring to the stage.
Event-driven solutions are not a panacea --- they are simply the most appropriate solution to several salient modern business problems where traditional approaches have shortcomings.
The thirst for realtime data, processes, and user experiences is not going away, with the audience's demands only growing over time. Event-driven APIs exist to address critical business requirements of realtime situation awareness for decision makers; extensibility of applications at scale; and responsive, engaging user interfaces. The value of events is growing as well, giving us valuable insight into previously opaque system behaviors.
Knowing how event-driven architectural approaches are a fit for specific use-cases of the business --- implementing them with care where appropriate --- offers distinct advantages over competitors who lack EDA competency. Even more to the point, it keeps pace with the pulse of the users.
"Moving to event-driven approaches shifts the complexity of orchestrating user interactions onto the producer --- but it's well worth it. Tools like Ably, built on top of AsyncAPI, are there precisely to take care of that burden and make the life of producers easier."\
--- Łukasz Górnicki, Maintainer and Dev Community Keeper at AsyncAPI Initiative
About Ably
Ably is an enterprise-grade pub/sub messaging platform. We make it easy to efficiently design, quickly ship, and seamlessly scale critical realtime functionality delivered directly to end-users. Every day, we deliver billions of realtime messages to millions of users for thousands of companies.
We power the apps that people, organizations, and enterprises depend on everyday like Lightspeed System's realtime device management platform for over seven million school-owned devices, Vitac's live captioning for 100s of millions of multilingual viewers for events like the Olympic Games, and Split's realtime feature flagging for one trillion feature flags per month.
We're the only pub/sub platform with a suite of baked-in services to build complete realtime functionality: presence shows a driver's live GPS location on a home-delivery app, history instantly loads the most recent score when opening a sports app, stream resume automatically handles reconnection when swapping networks, and our integrations extend Ably into third-party clouds and systems like AWS Kinesis and RabbitMQ.
With 25+ SDKs we target every major platform across web, mobile, and IoT. We are committed to the OpenAPI and AsyncAPI initiatives, as well as the Open Data Streaming Program (ODSP), so organizations can host and distribute open streaming data sources for free using the Ably Hub.
Our platform is mathematically modeled around Four Pillars of Dependability so we're able to ensure messages don't get lost while still being delivered at low latency over a secure, reliable, and highly available global edge network.
Try our APIs and see how we make the move to event-driven easy, or get in touch to learn more about how realtime, event-driven solutions can augment your SOA stack.
Top comments (1)
I enjoy reading your post. I learned a lot from it! city chevrolet