Let's say we have a blog website like Dev.to! So, every time a user publishes a blog, the server needs to update the user's blog count in the database.
With millions of users publishing blogs, API server has to compute and update the DB.
Isn't that too much of load for the API server ?
Why not let someone else do this job.
This is where Message Brokers come in. Who knows dev.to might also be using these
Message Brokers
Instead of the API server directly updating the database, it sends a message to the broker. The broker stores the message in a queue or stream, and worker components process it later.
So our worker component have two jobs in our scenerio
- Updating Count in DB
- Indexing
How Message Brokers Work
Message brokers can be implemented in two main ways:
1. Message Queues (e.g., RabbitMQ, SQS)
In a message queue, messages are stored in a queue and pulled by consumers (workers). For example, a worker might update the blog count in the database and perform indexing.
Once a message goes to one consumer it cannot go to another one.
Great! Whats the problem then ?
- What if the worker crashes after updating the database but before indexing? This can lead to inconsistencies and this is why we go for message streams
2. Message Streams (e.g., Kafka, Kinesis)
In message streams like Kafka or Kinesis, messages are loaded into a stream, and consumers process them at their own pace.
Unlike queues, messages are not pulled by consumers. Instead, consumers independently go through the stream and process data.
This approach ensures better fault tolerance and consistency, as each consumer processes messages independently.
Conclusion
Well this is high level architecture of things, I hope you found this helpful. Thanks for reading :)
Top comments (0)