Redis Pub/Sub vs. Redis Streams: Choosing the Right Solution

While working on building a Go binary tool ("Runner") for our product LiveAPI (the super convenient API doc generation tool), we needed to establish communication between LiveAPI's backend and Runner. We initially tried the Redis pub/sub mechanism but later realized it was insufficient for our use case.

The Problems with Redis Pub/Sub

Redis Pub/Sub follows a traditional publish-subscribe model: Publishers send messages to channels, and subscribers receive messages from the channels they are subscribed to.

This presented several problems for LiveAPI:

Duplicate Operations and Wasted Resources: If a user ran the same Runner on multiple devices, the subscription behavior of pub/sub would cause the same operations to run on multiple devices, creating duplication and wasting resources.

Memory and Data Loss: Pub/sub doesn't retain or persist messages, so if the server crashed or a job failed due to a panic, there was no way to restart the operations.

Lack of Acknowledgement: The LiveAPI backend had no indication that a job request from the consumer had reached the subscriber.

Redis Streams as the Solution

Redis Streams provide a more robust and feature-rich messaging system. They are essentially append-only logs where producers add messages, and consumers read them.

Redis Streams addresses the issues we encountered with Pub/Sub:

Avoids Duplicate Operations: Even if the same Runner is running on multiple devices, only one instance can read and acknowledge a message, preventing the duplicate operations that occurred with pub/sub.

Message Persistence: Redis Streams persist messages. If the server crashes, operations can be restarted with the same message.

Guaranteed Delivery and Acknowledgement: Redis Streams provides message acknowledgement, assuring the LiveAPI backend that job requests reach the intended subscriber.

Understand Your Usecase
Redis Streams was selected as a solution for LiveAPI due to its message persistence, guaranteed delivery, message acknowledgement, and ability to prevent duplicate operations.

While Pub/Sub is a simpler solution and better suited for real-time messaging, Streams offers more features and guarantees, making it ideal for use cases where data durability and reliability are critical.