Introduction
In a client-server architecture, the client can request a job to be done from the server by sending messages between each other.
Handling this communication can increase in complexity when you begin to manage the rate at which messages are sent, the number of requests a server can handle, or the response time which the client demands.
In this blog, we'll see one way to handle this intricacy.
What is Synchronous Processing?
In synchronous processing, a client sends a request to the server and waits for the server to complete its job and send back the response before the client can resume doing any other work.
This process is often referred to as a blocking request as the client gets blocked from doing any other work until a response is received.
What is Asynchronous Processing?
Asynchronous processing is the exact opposite of synchronous processing. Here, the client doesn't wait for a response after sending a request to the server and continues doing any other work.
This process is referred to as a non-blocking request as the execution thread of the client is not blocked. This allows systems to scale as more work can be done in a given amount of time.
Synchronous vs Asynchronous Processing
Synchronous requests block the execution thread of the client, forcing them to wait for the response to come before they can perform another action. On the other hand, asynchronous requests do not block and allow for more work to be done in a given time.
Since we do not know much time the request will take, it is difficult to build responsive applications with synchronous processing. The more blocking operations, the slower system becomes. With asynchronous processing, response time is quick as the client does not have to wait on the request.
Fault tolerance of asynchronous processing is higher than that of synchronous processing, as it is easy to build a retry mechanism when a request fails.
What are Message Queues?
A message queue is a component that buffers requests from one service and broadcasts asynchronously to another service.
Here, clients are the message producers, who send request messages to the queue instead of any server. They get an acknowledgment when the message is added to the queue, which enables them to continue with their other jobs without having to wait for the server.
Servers are known as message consumers and are served these messages from the queue based on the number of requests they can serve at any given time. This continues until all the messages in the queue have been served.
The two most common messaging queues are — RabbitMQ and Kafka.
Structure of Message Queues
A message queue is primarily a broker of messages between message producers and message consumers.
Each distinct entity in the messaging queue setup (producers, consumers, and queue) has a responsibility and they're decoupled from each other as much as possible.
The only contract between all entities is the messages for which the message queue facilitates the movement from producers to consumers.
In the following sections, we will discuss the responsibilities of each component and look at the various modes with which the message queue delivers a message to consumers.
Message Producers
Message producers initiate the asynchronous processing request. Producers have a responsibility to generate a valid message and publish it to the message queue. Messages submitted to the queue are then queued up and delivered to consumers to be processed asynchronously.
Producers communicate with message queues using the Advanced Message Queuing Protocol (AMQP).
Message Brokers
A message broker is simply just a queue. You can even implement a simple broker programmatically that buffers messages and sends them to consumers as and when needed.
Message brokers are the actual decoupling elements in the setup, sitting between and managing the process of communication between producers and consumers.
Because of their simplicity, brokers are optimized for high concurrency and throughput.
It is important to note that adding message brokers introduces an extra layer of complexity into your infrastructure and requires you to scale them as well. Brokers also have specific requirements and limitations when it comes to scalability.
Message Consumers
The main responsibility of consumers is to receive and process messages from the queue.
Most consumers are API services that perform that actual asynchronous processing.
Consumers can be implemented in different application languages or technologies and maintained independently from other components.
To achieve decoupling, consumers should know nothing about the producers. The only contract that should exist between the two is valid messages from the queue.
When properly decoupled, consumers can serve as independent service layers that can be used by both the message queue setup and other components in your infrastructure.
Consumer Communication Strategies
Message queues need to transmit messages down to consumers, depending on how application developers implement consumers, message queues have three distinct ways of delivering messages to the consumers:
- Pull Model
In this model, the consumer periodically checks the status of the queue. This is done at a scheduled interval programmed on the side of the consumer.
If there are messages found in the queue, the consumer picks them up until there are no more messages left to process, or when the 'N' number of messages has been consumed. This 'N' can be configured on the message broker.
- Push Model
Once a message is added, the consumer is notified and the message is then pushed down to it. Messages are pushed down to consumers at a rate at which the consumer can easily regulate.
- Subscription Model
In this model, consumers can subscribe to a topic. This publisher publishes a message to a topic rather than a queue. Each consumer connected to the broker maintains its private queue to receive messages from topics.
After the consumers subscribe to the topics and when a message is published to that topic, the message is cloned for each subscriber and added to the consumer’s private queue.
Comparing Different Message Brokers
As we have seen above, for asynchronous communication we usually need a message broker.
Below are the few considerations you have to look at when choosing a broker for managing your asynchronous operations:
- Scale: The number of messages sent per second in the system
- Data Persistency: The capability to recover messages
- Consumer Capability: The capability to manage one-to-one / one-to-many consumers
RabbitMQ
- Scale: Based on configuration and resources.
- Persistency: Both persistent and transient messages are supported.
- One-to-one vs One-to-many consumers: Both.
RabbitMQ supports all major languages, including Python, Java, .NET, PHP, Ruby, JavaScript, Go, Swift, and more.
Kafka
- Scale: Can send up to a million messages per second.
- Persistency: Yes.
- One-to-one vs One-to-many consumers: Only one-to-many
Kafka has managed SaaS on both Azure and AWS. Kafka also supports all major languages, including Python, Java, C/C++, Clojure, .NET, PHP, Ruby, JavaScript, Go, Swift, and more.
Redis
- Scale: Can send up to a million messages per second.
- Persistency: Not supported (it’s an in-memory database).
- One-to-one vs One-to-many consumers: Both.
Redis is a bit different from the other message brokers. Redis is an in-memory data store. Originally, Redis only supported one-to-one communication with consumers. However, since Redis 5.0 introduced the pub-sub, you can have one-to-many as another option.
Conclusion
In this blog, we covered how asynchronous processing is advantageous over its counterparts and how a message queue helps us achieve asynchronous processing along with keeping the different entities in its setup decoupled from each other.
We also covered some basic characteristics of the most commonly used message brokers: Redis, Kafka, and RabbitMQ.
Here's a bit more detailed instruction for selecting the right message broker to use according to different use cases:
Short-lived Messages: Redis
- Redis is good for short-lived messages where persistence isn’t required.
Large Amounts of Data: Kafka
- Kafka is good for storing a large amount of data for long periods. Kafka is also ideal for one-to-many use cases where persistency is required.
Complex Routing: RabbitMQ
- RabbitMQ is good for complex routing communication.
Happy coding! 💻
(If you find any doubts, updates, or corrections to improve this article, Feel free to share them in the comments) 😊
Starting out in web development?? 💻
Checkout ▶ HTML To React: The Ultimate Guide
This ebook is a comprehensive guide that teaches you everything you need to know to be a web developer through a ton of easy-to-understand examples and proven roadmaps
It contains 👇
✅ Straight to the point explanations
✅ Simple code examples
✅ 50+ Interesting project ideas
✅ 3 Checklists of secret resources
✅ A Bonus Interview prep
You can even check out a free sample from this book
and here's the link with 60% off on the original price on the complete book set ⬇
Top comments (3)
Subscription Model doesn't necessarily have to maintain private consumer queue. Take a look at Apache Pulsar where you have shared consumers meaning cursor is kept on the message system side but there can be many consumers that share the responsibility of reading the messages. So queue is not necessarily "private" per consumer but rather per subscription so system can track which messages were delivered per subscription to avoid double message processing by two different consumers in the same subscription. Also Apache Pulsar keeps messages in a queue even if they are "topics" so that's another part which can differ based on messaging system. They do have partitioned topics as well non-partitioned so some features might differ on the type of the topic but never the less I wanted to point out that there is some space for differences between what is subscription model.
I would rather say you have only two types - pull and push. Then subscription model is different feature of these system and can be divided in certain categories and discussed regardless of the communication strategy. I usually assume that subscription means push model as subscribers get notified as messages come in rather then checking periodically but in some cases you might keep track of message status on messaging system yet not notify consumers but let them ask for next message in line or allow replying / seeking previous messages. Basically subscription model is big enough to be discussed on it's own in my opinion.
Why not MQTT? Is not only for IoT very lightweight and reliable. And i would include here cloud solutions like AWS SQS or GCP Pub/Sub :)
I would not include cloud specific solutions as they work for "vendor lock in" in a way. To use them you have to use their cloud solution. On the other hand Kafka, RabbitMQ, RocketMQ, Pulsar, EMQx and such are deployable on any platform as "manually" managed VMs (containers) or by using Kubernetes helm (still kind of a manual management but less). Given that those can be deployed on-prem or in any cloud there's not so much to gain from building against cloud specific solutions. In fact building against cloud specific solutions is the new way of using compiled languages against specific OS and architectural type in general.
On the other hand MQTT may be really good but it's not great for each use case yet you can use it with Kafka and Pulsar for example - they have modules / plug-ins to communicate with that specific protocol in mind. And I would argue way of forming and passing data is not important for general discussion, you pick the one that suits, you and most of these although having their own protocol yet depend on general purpose data transfer and can easily be modified to support another protocol. If you use something that can use multiple protocols you gain and more and more these tools support such scenarios.