Introduction
Have you ever found yourself working on a design for an Event Driven Architecture (EDA)? You're not alone. More and more solutions are adopting EDA, and one of the challenges you might encounter is how to prioritize messages within such an architecture. It can indeed be a challenge.
Unlike non-EDA systems, where most of the data is readily available, data in EDA systems is almost always changing, making priority processing a complex task. What may be a priority at a given time T might not hold the same importance at time T+1. In these transient systems, the application processing the messages must adapt and implement some form of priority mechanism. While absolute priority processing may not be feasible in real-time event-driven systems, some level of prioritization can certainly be achieved.
This blog aims to share some common design patterns for implementing prioritized processing in EDA systems. It focuses on the consumer side of these architectures, as there are often limited or no controls over how messages are produced in a prioritized manner.
Approach 1: Source Separation: Different Sources for Priority vs Non-Priority Messages
It's the most trivial and simplistic implementation which can cater to implementing basic priority in processing records at consumer end.
Having dedicated input source/topic for priority messages keeps them separate from the bulk of non-priority messages and helps allocating dedicated processing capacity. We can spin up separate instances of consumer for dedicatedly processing priority messages.
High Level Design
PRO/CON Analysis
Pros
- Simple and Faster Implementation.
- Independent scaling based on traffic patterns for messages based on priority.
Cons
- Increased implementation complexity due to maintenance and processing from two different sources.
- Producer systems may not support the option of sending messages to two different sources.
Approach 2: Priority Queue: Using In-Memory Prioritization Framework
n systems where separate input sources for different priority messages are not feasible, consumer systems can implement an in-memory prioritization mechanism. A commonly used data structure for this purpose is Priority Queues. This approach allows the consumer system to adjust prioritization criteria according to their needs while simplifying the source system by maintaining a single input source.
High Level Design
To optimize Priority Queue implementation and reduce the frequency of re-arranging, records can be read in batches from the input source. In EDA systems using Kafka, this capability is inherently supported by the platform's consumer APIs, with parameters like max.poll.records.
Time Complexity of Priority Queue
The time complexity of Priority Queue for insertion(enqueue) and deletion (dequeue) methods, is O(log(n)).
For removal and contains methods, the time complexity is linear. Methods that retrieve elements from the Priority Queue have constant time complexity. By default, Priority Queue elements are naturally ordered. To change the ordering, a comparator can be specified during the creation of the Priority Queue object, which the Priority Queue will then use to order its elements. Several commercial products, such as Redis, offer this feature for reuse.
Deciding criteria for priority queue implementation
- Based on attribute in incoming payload
- Predefined set of rules which application can implement.
PRO/CON Analysis
Pros
- Relatively simple implementation. In Memory Caching systems like Redis has this supported.
- Prioritization Criteria can be customized at consumer end.
Cons
- In memory Queue is Vulnerable to System failures.
- It's an additional Hop in system and introduces some additional processing time.
- Additional Overhead for maintaining In-memory queues.
Approach 3: Database: Using DB for Storing and Priority Processing
Using databases to store and fetch messages based on priority can be an alternative for systems seeking more robust and scalable prioritization. Leveraging a database for storage ensures that the system can handle failures more reliably. Records can be retrieved based on various prioritization criteria using an SQL interface.
High Level Design
PRO/CON Analysis
Pros
- Robust failure mechanism handling leveraging DB system.
- High Flexibility in customizing/changing Prioritization Criteria. Prioritization related SQL queries can be stored and updated outside of application scope On-Demand basis.
Cons
- Additional Overhead with DB. SYNC time between Input Source and DB records may introduce some latency in overall processing.
- Additional Cost implication related to Database.
Conclusion
Implementing priority in real-time event-based systems remains an architectural challenge, and the design patterns discussed above are some general approaches to address this need. Absolute priority implementation is elusive in these systems due to constantly changing data, necessitating some form of time window or slicing to define the scope of prioritization.
As more systems adopt real-time EDA processing, the demand for priority implementation is growing, and various approaches have evolved across the industry to meet this need. The final choice for any implementation depends on finding the optimal balance between the pros and cons of each approach.
This blog aimed to share knowledge on possible architectures for priority implementation in real-time event-driven systems, based on my years of industry experience with similar use cases. I hope this provides valuable insights and thought processes for teams and applications with similar needs.
Top comments (0)