Redis Streams: A Comprehensive Guide

Redis Streams is getting popular nowadays. Let me give a comprehensive guide about it.

Redis Streams: Introduction

Redis Streams is a powerful feature introduced in Redis 5.0 that enables real-time data processing and event-driven architectures. It provides a log-like data structure that allows multiple consumers to read from the same data stream efficiently. Redis Streams is designed for high-throughput applications requiring message queues, distributed event processing, and data streaming solutions.

This guide explores how Redis Streams work, their key features, and practical use cases.

---

1. What Are Redis Streams?

Redis Streams is a data structure that stores a sequence of messages with unique IDs, preserving insertion order. It supports both pub/sub-like streaming and consumer groups, making it ideal for real-time analytics, logging, and event-driven applications.

Each message in a stream consists of:

• A unique ID (timestamp-based, e.g., 1678456789123-0).
• One or more key-value pairs as the message content.

---

2. Creating and Adding Data to a Stream

To create a new stream and add data, use the XADD command.

Example: Adding messages to a stream

XADD mystream * temperature 22.5 humidity 60
XADD mystream * temperature 23.0 humidity 58

Explanation:

• XADD adds a new entry to mystream.
• * generates a unique timestamp-based ID automatically.
• temperature and humidity are fields in the message.

---

3. Reading Data from a Stream

Use XRANGE or XREAD to fetch messages.

Reading All Messages

XRANGE mystream - +

Output:

1) 1678456789123-0
   1) "temperature" 2) "22.5"
   3) "humidity" 4) "60"
2) 1678456789135-0
   1) "temperature" 2) "23.0"
   3) "humidity" 4) "58"

• XRANGE mystream - + fetches all entries (- for start, + for end).

Reading Latest Messages

To continuously fetch new messages, use XREAD.

XREAD BLOCK 5000 STREAMS mystream $

• BLOCK 5000 waits for new data (timeout in ms).
• $ starts from the latest entry.

---

4. Stream Trimming: Managing Memory Usage

Streams can grow indefinitely, but Redis provides XTRIM to limit their size.

XTRIM mystream MAXLEN 1000

• Retains only the latest 1000 messages.
• Prevents excessive memory usage in high-throughput systems.

---

5. Consumer Groups: Parallel Processing

Redis Streams supports consumer groups, allowing multiple clients to process messages in parallel.

Creating a Consumer Group

XGROUP CREATE mystream mygroup $

• mygroup is the group name.
• $ starts from the latest message.

Adding Consumers

Consumers in the group can read messages using XREADGROUP.

XREADGROUP GROUP mygroup consumer1 COUNT 2 STREAMS mystream >

• consumer1 reads up to 2 messages.
• > fetches only pending messages.

Acknowledging Messages

After processing a message, acknowledge it with XACK to remove it from the pending list.

XACK mystream mygroup 1678456789123-0

---

6. Use Cases for Redis Streams

• Real-time Analytics: Collecting sensor data, tracking website events.
• Message Queues: Distributing tasks among workers.
• Event Sourcing: Logging events in applications.
• Log Aggregation: Storing and processing application logs.

---

7. Redis Streams vs. Kafka

Feature	Redis Streams	Apache Kafka
Message Storage	In-memory	Disk-based
Consumer Model	Push & Pull	Pull-based
Scalability	Limited	High
Persistence	Optional	Persistent

Redis Streams is lightweight and ideal for small to mid-sized applications, whereas Kafka is suited for large-scale distributed systems.

---

Redis Streams: Conclusion

Redis Streams provides a robust and efficient solution for real-time messaging and event-driven architectures. With its support for consumer groups, trimming, and persistent event storage, Redis Streams is a great choice for applications requiring high-speed data processing.

Whether you’re building real-time analytics dashboards, distributed job queues, or event-driven microservices, Redis Streams offers a flexible and scalable solution.