Go service with Consul integration

In the world of stateless microservices, which are usually written in Go, we need to discover them. This is where Hashicorp’s Consul helps. Services register within Consul so other services can discover them via simple DNS or HTTP queries.

Go has a Consul client library, alas, I didn’t see any real examples of how to integrate it into your services. So here I’m going to show you how to do exactly this.

I’m going to write a service that will serve at some HTTP endpoint and will serve key-value data – I believe this resembles a lot of existing microservices that people write these days. Ours is called webkv and it’s on Github. Choose the “v1” tag and you’re good to go.

This service will register itself in Consul with TTL check that will, well, check internal health status and send a heartbeat like signals to Consul. Should Consul not receive a signal from our service within a TTL interval it will mark it as failed and remove it from queries results.

Side note: Consul has also simple port checks when Consul agent will judge the health of the service based on the port availability. While it’s much simpler, e.g. you don’t have to add anything to your code, it’s not that powerful as a TTL check. With TTL checks you can inspect the internal state of your service which is a huge advantage in comparison with simple availability – you can accept queries but your data may be stale or invalid. Also, with TTL checks service status can be not only in binary state – good/bad – but also with a warning.

All right, to the point! The “v1” version of webkv uses only the standard library and the bare minimum of dependencies like Redis client and Consul API lib. Later I’m going to extend it with other niceties like Prometheus integration, structured logging, and sane configuration management.

Basic Web service

Let’s start with a basic web service that will serve key-value data from Redis.

First, parse port, ttl, and addrs commandline flags. The last one is the list of Redis addresses separated with ;.

func main() {
    port := flag.Int("port", 8080, "Port to listen on")
    addrsStr := flag.String("addrs", "", "(Required) Redis addrs (may be delimited by ;)")
    ttl := flag.Duration("ttl", time.Second*15, "Service TTL check duration")
    flag.Parse()

    if len(*addrsStr) == 0 {
        fmt.Fprintln(os.Stderr, "addrs argument is required")
        flag.PrintDefaults()
        os.Exit(1)
    }

    addrs := strings.Split(*addrsStr, ";")

Now, we create a service that should implement Handler interface and launch it.

    s, err := service.New(addrs, *ttl)
    if err != nil {
        log.Fatal(err)
    }
    http.Handle("/", s)

    l := fmt.Sprintf(":%d", *port)
    log.Print("Listening on ", l)
    log.Fatal(http.ListenAndServe(l, nil))

Nothing fancy here. Now let’s look at the service itself.

import (
    "time"

    "github.com/go-redis/redis"
)

type Service struct {
    Name        string
    TTL         time.Duration
    RedisClient redis.UniversalClient
}

The Service is a type that holds a name, TTL and Redis client handler. It’s instantiated like this:

func New(addrs []string, ttl time.Duration) (*Service, error) {
    s := new(Service)
    s.Name = "webkv"
    s.TTL = ttl
    s.RedisClient = redis.NewUniversalClient(&redis.UniversalOptions{
        Addrs: addrs,
    })

    ok, err := s.Check()
    if !ok {
        return nil, err
    }

    return s, nil
}

Check method issues PING Redis command to check if we’re ok. This will be used later with Consul registration.

func (s *Service) Check() (bool, error) {
    _, err := s.RedisClient.Ping().Result()
    if err != nil {
        return false, err
    }
    return true, nil

And now the implementation of ServeHTTP method that will be invoked for request processing:

func (s *Service) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    status := 200

    key := strings.Trim(r.URL.Path, "/")
    val, err := s.RedisClient.Get(key).Result()
    if err != nil {
        http.Error(w, "Key not found", http.StatusNotFound)
        status = 404
    }

    fmt.Fprint(w, val)
    log.Printf("url=\"%s\" remote=\"%s\" key=\"%s\" status=%d\n",
        r.URL, r.RemoteAddr, key, status)
}

Basically, what we do is retrieve URL path from the request and use it as a key for Redis “GET” command. After that, we return the value or 404 in case of an error. Last, we log the request with a quick and dirty structured logging message inlogfmt format.

Launch it:

$ ./webkv -addrs 'localhost:6379'
2017/12/13 21:44:15 Listening on :8080

Query it:

$ curl 'localhost:8080/blink'
182

And see the log message:

2017/12/13 21:44:29 url="/blink" remote="[::1]:35020" key="blink" status=200

Consul integration

Now let’s make our service discoverable via Consul. Consul has simple HTTP API to register services that you can employ directly via “net/http” but we will use its Go library.

Consul Go library doesn’t have examples, BUT, it has tests! Tests are nice not only because it gives you confidence in your lib, approval for the sanity of your code structure and API and, finally, a set of usage examples. Here is an example from Consul API test suite for service registration and TTL checks.

Looking at these tests, we can tell that we interact with Consul by creating a Client and then getting a handle for the particular endpoint like/agent or /kv. For each endpoint, there is a corresponding Go type. Agent endpoint is responsible for service registration and sending health checks. To store an Agent handle we extend our Service type with a new pointer:

import (
    consul "github.com/hashicorp/consul/api"
)

type Service struct {
    Name        string
    TTL         time.Duration
    RedisClient redis.UniversalClient
    ConsulAgent *consul.Agent
}

Next, in the Service “constructor” we add the creation of Consul agent handle:

func New(addrs []string, ttl time.Duration) (*Service, error) {
    ...
    c, err := consul.NewClient(consul.DefaultConfig())
    if err != nil {
        return nil, err
    }
    s.ConsulAgent = c.Agent()

Next, we use the agent to register our service:

    serviceDef := &consul.AgentServiceRegistration{
        Name: s.Name,
        Check: &consul.AgentServiceCheck{
            TTL: s.TTL.String(),
        },
    }

    if err := s.ConsulAgent.ServiceRegister(serviceDef); err != nil {
        return nil, err
    }

The key thing here is the Check part where we tell Consul how it should check our service. In our case, we say that we ourselves will send heartbeat-like signals to Consul so that it will mark our service failed after TTL. Failed service is not returned as part of DNS or HTTP API queries.

After service is registered we have to send a TTL check signal with Pass, Fail or Warn type. We have to send it periodically and in time to avoid service failure by TTL. We’ll do it in a separate goroutine:

go s.UpdateTTL(s.Check)

UpdateTTL method uses time.Ticker to periodically invoke the actual update function:

func (s *Service) UpdateTTL(check func() (bool, error)) {
    ticker := time.NewTicker(s.TTL / 2)
    for range ticker.C {
        s.update(check)
    }
}

check argument is a function that returns a service status. Based on its result we send either pass or fail check:
go

func (s *Service) update(check func() (bool, error)) {
    ok, err := check()
    if !ok {
        log.Printf("err=\"Check failed\" msg=\"%s\"", err.Error())
        if agentErr := s.ConsulAgent.FailTTL("service:"+s.Name, err.Error()); agentErr != nil {
            log.Print(agentErr)
        }
    } else {
        if agentErr := s.ConsulAgent.PassTTL("service:"+s.Name, ""); agentErr != nil {
            log.Print(agentErr)
        }
    }
}

Check function that we pass to goroutine is the one we used earlier on creating service, it just returns bool status of Redis PING command.

And that’s it! This is how it all works together:

• We launch the webkv
• It connects to Redis and starts serving at given port
• It connects to Consul agent and register service with TTL check
• Every TTL/2 seconds we check service status by PINGing Redis and send Pass check
• Should Redis connectivity fail we detect it and send a Fail check that will remove our service instance from DNS and HTTP query to avoid returning errors or invalid data

To see it in action you need to launch a Consul and Redis. You can launch Consul with consul agent -dev or start a normal cluster. How to launch Redis depends on your distro, in my Fedora, it’s just systemctl start redis.

Now launch the webkv like this:

$ ./webkv -addrs localhost:6379 -port 8888
2017/12/14 19:00:29 Listening on :8888

Query the Consul for services:

$ dig +noall +answer @127.0.0.1 -p 8600 webkv.service.dc1.consul
webkv.service.dc1.consul. 0 IN A 127.0.0.1

$ curl localhost:8500/v1/health/service/webkv?passing
[
    {
        "Node": {
            "ID": "a4618035-c73d-9e9e-2b83-24ece7c24f45",
            "Node": "alien",
            "Address": "127.0.0.1",
            "Datacenter": "dc1",
            "TaggedAddresses": {
                "lan": "127.0.0.1",
                "wan": "127.0.0.1"
            },
            "Meta": {
                "consul-network-segment": ""
            },
            "CreateIndex": 5,
            "ModifyIndex": 6
        },
        "Service": {
            "ID": "webkv",
            "Service": "webkv",
            "Tags": [],
            "Address": "",
            "Port": 0,
            "EnableTagOverride": false,
            "CreateIndex": 15,
            "ModifyIndex": 37
        },
        "Checks": [
            {
                "Node": "alien",
                "CheckID": "serfHealth",
                "Name": "Serf Health Status",
                "Status": "passing",
                "Notes": "",
                "Output": "Agent alive and reachable",
                "ServiceID": "",
                "ServiceName": "",
                "ServiceTags": [],
                "Definition": {},
                "CreateIndex": 5,
                "ModifyIndex": 5
            },
            {
                "Node": "alien",
                "CheckID": "service:webkv",
                "Name": "Service 'webkv' check",
                "Status": "passing",
                "Notes": "",
                "Output": "",
                "ServiceID": "webkv",
                "ServiceName": "webkv",
                "ServiceTags": [],
                "Definition": {},
                "CreateIndex": 15,
                "ModifyIndex": 141
            }
        ]
    }
]

Now if we stop the Redis we’ll see the log messages

...
2017/12/14 19:29:19 err="Check failed" msg="EOF"
2017/12/14 19:29:27 err="Check failed" msg="dial tcp [::1]:6379: getsockopt: connection refused"
...

and that Consul doesn’t return our service:

$ dig +noall +answer @127.0.0.1 -p 8600 webkv.service.dc1.consul
$ # empty reply

$ curl localhost:8500/v1/health/service/webkv?passing
[]

Starting Redis again will make service healthy.

So, basically this is it – the basic Web service with Consul integration for service discovery and health checking. Check out the full source code at github.com/alexdzyoba/webkv. Next time we’ll add metrics export for monitoring our service with Prometheus.