How to Build a Concurrent Key-Value Store in Go

Key-Value stores are one of the simplest yet most effective forms of data storage used in modern applications.

In this tutorial, we will walk through building NanoKV, a minimalist in-memory key-value store written in Go, using the Fiber web framework.

By the end of this tutorial, you’ll understand:

• How key-value stores work under the hood
• Implementing basic CRUD operations in Go
• Using concurrency and TTL (Time-To-Live) for ephemeral data storage
• Running the application in a Docker container

Prerequisites

Before we dive in, ensure you have the following:

• Go installed (Download here)
• Basic understanding of Go programming
• Docker installed (optional, for containerization)

Getting Started

Let’s start by setting up our project structure.

1. Initialize a Go Project

First, create a new project directory and initialize a Go module:

mkdir NanoKV && cd NanoKV
go mod init NanoKV

2. Install Dependencies

We’ll use Fiber for building the API. Install it using:

go get github.com/gofiber/fiber/v2

3. Project Structure

Your project should have the following structure:

NanoKV/
│── kvstore/
│   └── kvstore.go
│── main.go
│── go.mod
│── go.sum
│── Dockerfile

Implementing the Key-Value Store

1. Defining the Store Structure

Create a new file kvstore/kvstore.go and define the data structure:

package kvstore

import (
    "sync"
    "time"
)

type Data struct {
    value      string
    expiration time.Time
}

type KeyValueStore struct {
    mu   sync.RWMutex
    data map[string]Data
}

func NewKeyValueStore() *KeyValueStore {
    return &KeyValueStore{
        data: make(map[string]Data),
    }
}

we use a map to store the actual key-value pairs. A map in Go is a built-in data structure that allows us to associate keys with values efficiently. However, since our key-value store is meant to be accessed concurrently (multiple goroutines may try to read or write to it at the same time), we need a mechanism to ensure safe concurrent access.

This is where sync.RWMutex comes in. It is a read-write mutual exclusion lock that allows multiple goroutines to read data simultaneously but ensures that only one goroutine can write at a time. This helps prevent race conditions and maintains data consistency.

• sync.RWMutex has two primary operations:
• RLock() and RUnlock() are used for read operations, allowing multiple readers to access the data simultaneously.
• Lock() and Unlock() are used for write operations, ensuring that only one goroutine can modify the data at any given time.

By wrapping our key-value store operations (such as Set, Get, and Delete) with these locks, we guarantee thread safety while maintaining high performance for concurrent reads.

2. Implementing CRUD Operations with TTL Support

Next, we implement methods to set, get, and delete key-value pairs.

func (kv *KeyValueStore) Set(key, val string, ttl time.Duration) {
    kv.mu.Lock()
    defer kv.mu.Unlock()

    kv.data[key] = Data{
        value:      val,
        expiration: time.Now().Add(ttl),
    }
}

func (kv *KeyValueStore) Get(key string) (string, bool) {
    kv.mu.Lock()
    defer kv.mu.Unlock()
    val, ok := kv.data[key]

    // Check for Item Expiry
    if val.expiration.IsZero() || time.Now().Before(val.expiration) {
        return val.value, ok
    }

    // Delete if Expired
    delete(kv.data, key)
    return "", false
}

func (kv *KeyValueStore) Delete(key string) bool {
    kv.mu.Lock()
    defer kv.mu.Unlock()

    _, ok := kv.data[key]
    if !ok {
        return false
    }
    delete(kv.data, key)
    return true
}

• Set Method: Stores a key-value pair with an expiration time. If the TTL (time-to-live) is set, it calculates the expiration timestamp. The function locks the store before writing and unlocks after completion.
• Get Method: Retrieves a value from the store. If the key exists and has not expired, it returns the value. If expired, it removes the key and returns false.
• Delete Method: Removes a key-value pair from the store. If the key exists, it deletes it and returns true; otherwise, it returns false.
• Concurrency Handling: A read-write mutex (sync.RWMutex) ensures thread safety by preventing race conditions when multiple goroutines access the store.

Exposing an HTTP API with Fiber

To make our key-value store accessible via HTTP, we set up a simple REST API using Fiber.

1. Setting Up the HTTP Server

Create a main.go file and add the following:

package main

import (
    "time"
    "NanoKV/kvstore"
    "github.com/gofiber/fiber/v2"
)

func main() {
    app := fiber.New()
    kv := kvstore.NewKeyValueStore()

    app.Get("/", func(c *fiber.Ctx) error {
        return c.SendString("This is a Simple Key-Value store like Redis in Go.")
    })

    app.Get("/get/:key", func(c *fiber.Ctx) error {
        key := c.Params("key")
        value, ok := kv.Get(key)
        if !ok {
            return c.SendString("The Key " + key + " doesn't exist")
        }
        return c.SendString("The Key " + key + " has Value " + value)
    })

    app.Post("/set/:key/:value", func(c *fiber.Ctx) error {
        key := c.Params("key")
        value := c.Params("value")
        kv.Set(key, value, 10*time.Minute)
        return c.SendString("Key " + key + " Value " + value)
    })

    app.Delete("/delete/:key", func(c *fiber.Ctx) error {
        key := c.Params("key")
        ok := kv.Delete(key)
        if !ok {
            return c.SendString("The Key " + key + " doesn't exist")
        }
        return c.SendString("Successfully Deleted!!")
    })

    app.Listen(":3000")
}

• GET /get/:key: Fetches the value of a given key if it exists; otherwise, returns a message indicating the key does not exist.
• POST /set/:key/:value: Stores a key-value pair with a default TTL of 10 minutes.
• DELETE /delete/:key: Deletes a key from the store if it exists.

Running the Server

Run the application:

go run main.go

Test it using curl:

curl -X POST http://localhost:3000/set/foo/bar
curl -X GET http://localhost:3000/get/foo
curl -X DELETE http://localhost:3000/delete/foo

Containerizing the Application with Docker

We can containerize our Go application using the following Dockerfile:

FROM golang:alpine

WORKDIR /app

COPY go.mod ./
COPY go.sum ./
RUN go mod download

COPY . ./

RUN go build -o NanoKV .

EXPOSE 3000

CMD ["/app/NanoKV"]

Building and Running the Docker Container

docker build -t nanokv .
docker run -p 3000:3000 nanokv

You can get the complete code here.

Conclusion

We have successfully built a lightweight in-memory key-value store in Go, complete with:

- Basic CRUD operations
- Concurrency handling
- TTL-based expiration
- An HTTP API using Fiber
- Containerization with Docker

You can further enhance this by adding features like persistent storage, distributed setup, or enhanced performance optimizations.

That’s it for this Tutorial. If this blog was helpful to you do share it and to get more information about Golang concepts, projects, etc. and to stay updated on the Tutorials do follow Siddhesh on Twitter and GitHub.

Until then Keep Learning, Keep Building 🚀🚀

Comments

[David Sugar]

I like that this is using fiber because fiber is also my choice for backend web services. It is also a use that shows the simplicity of go.