Go and Redis, Better Together 🤝

I recently presented this talk at the Conf42 Golang 2023 and I thought it might be a good idea to turn it into a blog post for folks who don't want to spend 40+ mins watching the talk (it's ok, I understand 😉) or just staring at slides trying to imagine what I was saying.

So here you go....

By the way, you are still welcome to watch the talk or download the slides! 🙌 There are a lot of great talks that you can get from this playlist.

This talk was geared towards folks who are looking to get started with Redis and Go. Or perhaps you are already experienced with both these topics – in that case, it might be a good refresher!

To that extent, I had a very simple agenda.

• I started off by setting the context about Redis and Go
• Provide an overview of Go and Redis ecosystem, including the client options you’ve got..
• Followed by some hand-on stuff and,
• Wrap with some tips/tricks and resources.

I love Redis and Go!

Since its release in 2009, it did not take Redis too long to win the hearts and minds of the developer community! As per db-engines trends statistics, Redis has been topping the charts since 2013. And on Stack Overflow annual survey, it’s been voted most loved database for 5 years in a row.

Go has become the language of the Cloud. It powers many cloud-native projects (apparently, 75% of CNCF projects are written in Go) including Docker, Kubernetes, Prometheus, Terraform etc. In fact there many databases written in Go - like Influxdb, etcd, Vitess, TiDB etc.

Go also caters to a wide variety of general purpose use cases:

• Web apps and APIs,
• Data processing pipelines,
• Infrastructure as code,
• SRE/DevOps solutions,
• Command line apps (this is a really popular one!)
• and more...

No wonder Go has become such a popular programming language!

Now you might be thinking - "Hey Go is down at the bottom." But if you notice carefully, it is the only statically-typed lang after Rust (of course there are C# and Kotlin down there as well) And this is from 2022 - if you look at data from 2021 to 2018, you will notice that Go has maintained its top 5 spot.

Go and Redis have a few things in common, but to me "Simplicity" is the one that really stands out to me.

Simplicity

Redis is a key value store, but the values can be any of these data structures that you see. These are all data structures that we as developers use every day - lists, sets, maps etc. Redis just feels like an extension to these core programming language constructs.

With Go, this comes in various forms…

• Excellent tooling,
• Or a comprehensive standard library,
• Easy to use concurrency primitives...

And sometimes it's in the form of not bloating the language with unnecessary features. To cite an example, it took a while for generics to be added to the language. Now I am not trying to trick you into thinking that Go is simple.. Or for that matter any programming language is simple.

But with Go, the goal is to give you simple components to help build complex things, and hide complexity behind a simple facade.

And there are folks who have explained this in great detail (and much better than I can!). I would really encourage you to check out this talk by Rob Pike (and the slides), the co-creator of Go (its from 2015 but still very much applicable to the essence and spirit of Go).

Redis 101

Quick intro to Redis (some of the key points):

• Data structure server - At its core, Redis is nothing but a key value store, where the key can be string or even a binary. The important thing to note is that as far as the value is concerned, you can choose from a variety of data structures such as strings,hashes, lists, sets etc.
• Redis is not just a cache - It’s a really solid messaging platform as well
• HA - You can configure redis to be highly avaialbe by setting up primary-replica replication, or take it a step further with Redis Cluster.
• Persistence: Redis is primarily in-memory but you can configure it to persist to disk as well. But there are solutions like Amazon MemoryDB that can actually take it a notch further (thanks to it's distributed transactional log)
• Since its open-source and wildy popular, you can get offerings from pretty much every cloud provider – big or small or even run it on Kubernetes, on cloud, on-prem or hybrid mode. If you want to put Redis in production, you can be rest assured that there are no dearth of options for you to run and operate it.

Redis data types

What you see here is a list of core data structures:

• A string seems really simple, but are quite powerful. They can be used for something as simple as storing a key-value pair to implementing advanced patterns like distributed locking and rate-limiting
• A hash is similar to a map in Java, or dictionary in Python. It is used to store object like structures like user profile, customer info etc.
• A set behaves like its mathematical version - helps you maintain unique set of items along with ability to list them, count them, execute unions, intersection and so on.
• Think of sorted sets are like this big brother to a set - makes it possible to implement things like leaderboards which is very useful in areas like gaming. For e.g. you can store player scores in a sorted set and when you need to get the top 10 players, you can simply invoke specific sorted set commands and get that data. The beauty is that sorting happens on the database side, so there is no client side logic you need to apply.
• Lists are a really versatile data structure as well. You can use them to store many things, but using it as a worker queue is very common. There are popular open source solution such as sidekiq and celery that already support Redis as a backend for job queuing solutions.
• Redis streams (added in Redis 5) is used for streaming use-cases
• And also ephemeral messaging with pub/sub - it's a publish-broadcast mechanism where you can send/receive messages to/from channels.
• There is also Geospatial data structure and a really cool one called Hyperloglog which is an alternative to a traditional set. It can store millions of items while optimizing for data storage and you can count the number of unique items with really high accuracy.

Go clients for Redis

👆🏼👆🏼 These are the most popular Go clients for Redis.

go-redis is by far the most popular client. It has what you’d expect.. Features, decent documentation, active community. Moving this under the official Redis GitHub org is just icing on the cake!

redigo is a fairly stable and tenured client that supports all the standard Redis data types and features such as transactions, pipelining etc.
It is also used to implement other Go client libraries such as redisearch-go and Redis TimeSeries Go client.
That being said, its API is a bit too flexible. While some may prefer that, but I feel that it’s not a good fit when I am using a type-safe language like Go (that’s just my personal opinion).

But the biggest drawback to me is that it does not support Redis Cluster!

rueidis is relatively new (at the time of writing) but quickly evolving client library. It supports RESP3 protocol, client-side caching and supports a variety of Redis Modules. As far as the API is concerned, this client adopts an interesting approach. It provides a Do function as well (like redigo client), but the way it creates the command is via a builder pattern - this retains strong type checking (unlike redigo).

To be honest with you, I haven't used this client a lot, and its looks like its packed with a lot of features. So, I cant complain much as of now!

For those who are looking for deeper performance numbers – this benchmark comparison with the go-redis library might be interesting.

Client ecosystem

Now lets take a look from an ecosystem perspective. To clarify, the point here is not chest thumping based on GitHub stars - its just to give you a sense of things.

For folks not using Go and Redis, the popularity of the Go client might come as a surprise. Java workloads form a huge chunk of the Redis workloads and Jedis is the bread and butter client when it comes to Java apps with Redis (and it's pretty old). But I was surprised to redisson topping the charts, followed by go-redis (yay!), (two) Nodejs clients, Python and finally back to Java.

Another thing to note is that I was looking for more than 10000 stars. At the time of writing, the phpredis client was close to 9600 stars.

Time for some practical stuff....

Demos

During the demo, I covered:

A walk though of the Go Redis client:

• Basics such as connecting to Redis
• Using common data types like string with TTL
• Use hash and struct support in go-redis
• How to use set as well as pipelining technique.

Hyperloglog and how does it differ from a Set:

func hllandset() {
    pipe := client.Pipeline()

    ips := []string{}

    for i := 1; i <= 10_00_000; i++ {
        ips = append(ips, fake.IP(fake.WithIPv4()))
    }

    pipe.SAdd(context.Background(), "set_of_ips", ips)
    pipe.PFAdd(context.Background(), "hll_of_ips", ips)

    pipe.Exec(ctx)

    //redis-cli MEMORY usage set_of_ips
    fmt.Println("no. of unique views (SCARD) -", client.SCard(ctx, "set_of_ips").Val())

    //redis-cli MEMORY usage hll_of_ips
    fmt.Println("no. of unique views (PFCOUNT) -", client.PFCount(ctx, "hll_of_ips").Val())

Chat application using pub/sub (below is a trimmed down version of the code):

package main

import (
    //omitted
)

var client *redis.Client
var Users map[string]*websocket.Conn
var sub *redis.PubSub
var upgrader = websocket.Upgrader{}

const chatChannel = "chats"

func init() {
    Users = map[string]*websocket.Conn{}
}

func main() {
    //...connect to redis (omitted)

    broadcast()

    http.HandleFunc("/chat/", chat)
    server := http.Server{Addr: ":8080", Handler: nil}

    //...start server (omitted)

    exit := make(chan os.Signal, 1)
    signal.Notify(exit, syscall.SIGTERM, syscall.SIGINT)
    <-exit

    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    //... clean up all connected connections, unsubscribe and shutdown server (omitted)
}

func chat(w http.ResponseWriter, r *http.Request) {
    user := strings.TrimPrefix(r.URL.Path, "/chat/")

    upgrader.CheckOrigin = func(r *http.Request) bool {
        return true
    }

    // 1. create websocket connection
    c, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Print("upgrade:", err)
        return
    }

    // 2. associate the user (name) with the actual connection
    Users[user] = c
    fmt.Println(user, "in chat")

    for {
        _, message, err := c.ReadMessage()
        if err != nil {
            //error handling and disconnect (omitted)
        }

        // 3. when a messages comes in via the connection, publish messages to redis channel
        client.Publish(context.Background(), chatChannel, user+":"+string(message))
    }
}

func broadcast() {
    go func() {
        sub = client.Subscribe(context.Background(), chatChannel)
        messages := sub.Channel()
        for message := range messages {
            from := strings.Split(message.Payload, ":")[0]
            // 3. if a messages is received on the redis channel, broadcast it to all connected sessions (users)
            for user, peer := range Users {
                if from != user {
                    peer.WriteMessage(websocket.TextMessage, []byte(message.Payload))
                }
            }
        }
    }()
}

Tips and tricks

In this section, I covered some of the points from Using Redis on Cloud? Here are ten things you should know.

These include:

• Connecting to Redis - common mistakes
• Scalability options - Vertical, Horizontal
• Using read-replicas
• Influence how your keys are distributed across a Redis cluster
• Execute bulk operations across Redis Cluster
• Sharded Pub/Sub

Resources

... and finally I wrapped up with some resources, including the Discord channel for Go Redis community.

Happy Building!