Daniel Kuroski

Posted on Jan 20

Building a Webhook payload delivery service in Go

#go #tutorial

This is a two part article, the full repo can be found here https://github.com/kuroski/go-webhook-delivers

Introduction

I was recently building an application that needed to listen to GitHub Webhooks.

For that, a "Payload URL" must be provided so GitHub can forward its events to this endpoint.

Once deployed and a public endpoint is available, it is a matter of only providing it.

But...

What to do during development?
How can I receive those Webhook events in my localhost?

There is no way I can just input http://localhost:3000 and expect it to work.

To develop this app locally, I needed a way to forward those events to the local environment.

In this tutorial, we will build a Webhook proxy URL.
It will forward events from some service (in this case GitHub) to your local application.

As a disclaimer, yes... There are plenty of ways to address this problem.

You can use services like smee.io (which is already suggested on GitHub tutorials).

Or even use something like ngrok to make the application securely available on its global edge in seconds.

But how cool it is to build this kind of service yourself, right?

What will we build?

This tutorial will consist of two applications

A CLI, which will run on our local machine, establishing a connection with the server and will be listening for events and forwarding them to a target local URL
A server that receives events from GitHub and forwards them to any listener (in this case, the CLI)

First, let's understand the big picture on how things will work.

We have four main players:

GitHub, that will produce the events in which will be dispatched to an endpoint
The server, will receive those events and forwards them to any listening client
The CLI, which will be running on your machine, it will subscribe for events on the server and forwards them
localhost, in this case, we will create a dummy server to receive the forwarded requests from the CLI

The CLI application will establish a connection with the server
GitHub will forward its events to an endpoint of our server
The request will be encoded
The encoded request will be forwarded to the CLI application
The data will be decoded
And finally, a new request will be made using the same parameters the GitHub Webhook request had sent initially
The connection between CLI and the server will close once we stop the CLI execution.

Server-sent events

This "connection" between the CLI and the Server happens in real time, but... How does this communication work?

There are plenty of ways to deal with "real-time". We could be using Polling, HTTP2 Push, Websockets, gRPC, etc.

But for this application we will be relying on SSE (server-sent events).

SSE is a pushing technology that enables pushing notification/message/events from the server to the client(s) via HTTP connection.
It has a simple protocol, and communication happens on a one-way connection, meaning that only the server sends events (unlike Websockets in which clients can also send events to the server), this makes it easier to reason and implement.

Starting out

First, let's create a basic structure for our project, then work out from it.

Create a folder for your project and start a new Go application

mkdir go-webhook-deliveries
cd go-webhook-deliveries
go mod init github.com/<your-username>/go-webhook-deliveries # I will be using `kuroski` as the name of my repo

We will be using a few packages, so let's install them

go get github.com/cenkalti/backoff/v5 github.com/lmittmann/tint github.com/tmaxmax/go-sse github.com/google/go-github/v67

Server

This will be a slim server with only two endpoints.

GET	/channel/{channel}	Starts sending incoming events to the client while logging any errors. CLI will use this endpoint to subscribe for messages.
POST	/channel/{channel}	GitHub will hit this endpoint with the events, the server will then encode the request and publish to its listeners.

It is not required to have a wildcard {channel}, feel free to work with a static endpoint if you don't need to have multiple entries.

First, we have our server main function, here we will be only instantiating its dependencies

// cmd/web/main.go

package main

import (
    "flag"
    "github.com/kuroski/go-webhook-deliveries/internal/server"
)

func main() {
    // grab the server port or default it to 3000 in case nothing is provided
    addr := flag.String("addr", ":3000", "HTTP network address")
    flag.Parse()

    // instantiate and start the server
    srv := server.NewServer(*addr)
    srv.Start()
}

Then, we can create the actual server implementation

// internal/server/server.go
package server

import (
    "context"
    "github.com/kuroski/go-webhook-deliveries/internal/logger"
    "github.com/tmaxmax/go-sse"
    "log/slog"
    "net/http"
    "os"
)

type Server struct {
    sseServer *sse.Server
    server    http.Server
    log       *slog.Logger
    ctx       context.Context
}

func NewServer(addr string) *Server {
    server := &Server{
        log: logger.NewLogger(),
        sseServer: &sse.Server{
            // this callback is called when an SSE session is started
            // here we set a subscription of the client for a given topic
            // if you decide that you don't need or want to handle wildcards, feel free to append the client to a static topic
            OnSession: func(s *sse.Session) (sse.Subscription, bool) {
                channel := s.Req.PathValue("channel")
                return sse.Subscription{
                    Client: s,
                    Topics: append([]string{channel}, sse.DefaultTopic),
                }, true
            },
        },
    }

    server.server = http.Server{
        Addr:     addr,
        Handler:  server.routes(),
        ErrorLog: slog.NewLogLogger(logger.NewLogger().Handler(), slog.LevelError),
    }

    return server
}

func (srv *Server) Start() {
    srv.log.Info("starting server", slog.String("addr", srv.server.Addr))

    err := srv.server.ListenAndServe()
    if err != nil {
        srv.log.Error(err.Error())
        os.Exit(1)
    }
}

Then, we need to define our routes

// internal/server/routes.go

package server

import (
    "net/http"
)

func (srv *Server) routes() http.Handler {
    mux := http.NewServeMux()
    mux.HandleFunc("GET /channel/{channel}", srv.subscribeToChannelHandler)
    mux.HandleFunc("POST /channel/{channel}", srv.forwardRequestToChannelHandler)
    return mux
}

And finally, the actual endpoint handlers

// internal/server/handlers.go

package server

import (
    "encoding/json"
    "fmt"
    "github.com/kuroski/go-webhook-deliveries/internal/model"
    "github.com/tmaxmax/go-sse"
    "io"
    "net/http"
    "time"
)

// this will handle the "GET /channel/{channel}" endpoint 
// it will establish the connection with the client and dispatch messages
// this will trigger the "OnSession" we configured on the server file, and will subscribe the client to a topic (in this case, the channel wildcard)
func (srv *Server) subscribeToChannelHandler(w http.ResponseWriter, r *http.Request) {
    srv.sseServer.ServeHTTP(w, r)
}

// this will handle the "POST /channel/{channel}" endpoint
// GitHub will send its events here, we will encode its data + request metadata and publish it to all channel listeners
func (srv *Server) forwardRequestToChannelHandler(w http.ResponseWriter, r *http.Request) {
    // first, we grab the "{channel}" wildcard value
    channel := r.PathValue("channel")

    // then, we read the Body content
    body, err := io.ReadAll(r.Body)
    defer r.Body.Close()
    if err != nil {
        srv.serverError(w, r, err)
        return
    }

    // we then wrap the received request into our own request struct (you can find its implementation below)
    // here we are wrapping everything needed to "re-create" the request later
    req := model.Request{
        Headers:   r.Header,
        Body:      body,
        Query:     r.URL.RawQuery,
        Timestamp: time.Now(),
    }

    // this is then encoded as JSON
    payload, err := json.Marshal(req)
    if err != nil {
        srv.serverError(w, r, err)
        return
    }

    // and then its content is wrapped over a SSE Message (which is a plain string)
    message := &sse.Message{}
    message.AppendData(string(payload))
    srv.log.Info("Hook received, forwarding the request", "req", req, "body", body)

    // finally, the message is published to all listeners to the channel
    if err = srv.sseServer.Publish(message, channel); err != nil {
        srv.serverError(w, r, err)
    }
    if _, err := fmt.Fprintf(w, "Message forwarded"); err != nil {
        srv.serverError(w, r, err)
    }
}

With that, we have our entire server configured and working.

Feel free to deploy this application if you have a preferred way.

If you don't know how to deploy a Go application, no worries, I will be writing in the appendices section how I did it.

Structs and utility functions

// internal/model/request.go

package model

import (
    "net/http"
    "time"
)

type Request struct {
    Headers   http.Header
    Body      []byte
    Query     string
    Timestamp time.Time
}

// internal/server/helpers.go

package server

import (
    "net/http"
    "runtime/debug"
)

func (srv *Server) serverError(w http.ResponseWriter, r *http.Request, err error) {
    var (
        method = r.Method
        uri    = r.URL.RequestURI()
        trace  = string(debug.Stack())
    )

    srv.log.Error(err.Error(), "method", method, "uri", uri, "trace", trace)
    http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
}

// internal/logger/logger.go

package logger

import (
    "github.com/lmittmann/tint"
    "log/slog"
    "os"
    "time"
)

func NewLogger() *slog.Logger {
    return slog.New(
        tint.NewHandler(os.Stdout, &tint.Options{
            Level:      slog.LevelDebug,
            TimeFormat: time.Kitchen,
            AddSource:  true,
        }),
    )
}

Creating the CLI

We have our server up and running, now we need to create the final piece.

Just like the server...

First, we have our main function, here we will be only instantiating its dependencies

// cmd/cli/main.go

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "flag"
    "github.com/cenkalti/backoff/v5"
    "github.com/kuroski/go-webhook-deliveries/internal/logger"
    "github.com/kuroski/go-webhook-deliveries/internal/model"
    "github.com/tmaxmax/go-sse"
    "net/http"
    "net/url"
    "os"
    "os/signal"
    "syscall"
)

func main() {
    // we need a source URL for listening to the events (this will be the in the end something like "https://my-app.com/channel/my-channel"
    source := flag.String("source", "", "The source server that we will listen for events")
    // and then the target URL, where we want the requests to be forwarded to, something like "http://localhost:3000"
    target := flag.String("target", "", "The target server that the request will be forwarded to")
    flag.Parse()

    log := logger.NewLogger()

    if *source == "" || *target == "" {
        log.Error("Required flags --source and --target must be set")
        flag.Usage()
        os.Exit(1)
        return
    }

    sourceURL, err := url.Parse(*source)
    if err != nil {
        log.Error("--source argument must be a valid url", "source", *source)
    }

    targetURL, err := url.Parse(*target)
    if err != nil {
        log.Error("--target argument must be a valid url", "target", *target)
    }

    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer cancel()

    // this is the most important part, we must create a SSE Client
    client := sse.DefaultClient

    // and then initialize a connection with the server
    r, _ := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL.String(), http.NoBody)
    conn := client.NewConnection(r)

    // finally, we need only to subscribe for messages sent from the server
    // every time a event occurs, this callback function will be called receiving an encoded "model.Request"
    conn.SubscribeMessages(func(event sse.Event) {
        // for the final steps, we need to Unmarshal the content
        var request model.Request
        if err := json.Unmarshal([]byte(event.Data), &request); err != nil {
            log.Error(err.Error())
            return
        }

        log.Info(
            "Message received",
            "headers",
            request.Headers,
            "query",
            request.Query,
            "timestamp",
            request.Timestamp,
            "body",
            request.Body[:100],
        )

        // and then we re-create the request to our target URL
        targetURL.RawQuery = request.Query
        req, err := http.NewRequest("POST", targetURL.String(), bytes.NewBuffer(request.Body))
        if err != nil {
            log.Error(err.Error())
            return
        }

        req.Header = request.Headers.Clone()
        res, err := client.HTTPClient.Do(req)
        if err != nil {
            log.Error(err.Error())
            return
        }

        log.Info("Request delivered successfully", "response", res)
    })

    // We are making use of "backoff" library to handle some edge cases.
    // After establishing a connection with the server, "tmaxmax/go-sse" library handles automatically the connection.
    // But it only happens after the first event is sent.
    // So, if there is no initial event for a while, we get a timeout and the application exits.
    // To avoid that situation, we use Exponential Backoff, meaning that in case of timeout, we avoid exiting the app, and try to establish a new connection instead after a period of time.
    _, err = backoff.Retry(ctx, func() (any, error) {
        log.Info("connecting sse, waiting for events from", "source", sourceURL.String(), "target", targetURL.String())
        if err := conn.Connect(); err != nil {
            log.Info(err.Error())
            return nil, err
        }

        return nil, nil
    }, backoff.WithBackOff(backoff.NewExponentialBackOff()))
    if err != nil {
        log.Error(err.Error())
    }
}

Testing out

If you finished everything and deployed your server, we can finally test our application.

If you don't know how to deploy a Go application, no worries, I will be writing in the appendices section how I did it.

First, go to your repository, and inside the "Settings > Webhooks" menu, let's create a new webhook

Provide your server URL and feel free to select which events you would like to trigger this webhook
I will be using the default which is triggered every time something is pushed to the project
Then, let's create a basic local server to receive the forwarded event from the CLI

// cmd/test/main.go

package main

import (
    "fmt"
    "github.com/google/go-github/v67/github"
    "net/http"
)

func main() {
    http.HandleFunc("/push-event", func(w http.ResponseWriter, r *http.Request) {
        payload, _ := github.ValidatePayload(r, nil)
        evt, _ := github.ParseWebHook(github.WebHookType(r), payload)
        pushEvt := evt.(*github.PushEvent)
        commit := pushEvt.Commits[0]
        fmt.Println("----- Webhook received", *commit.Message, *commit.Timestamp)
    })
    http.ListenAndServe(":3000", nil)
}

This should be only for testing the forwarded messages, so we will log only the commit message and timestamp.

Then, run the test server

go run ./cmd/test

Finally, let's run the CLI application

go run ./cmd/cli --source=https://my-server.com/channel/my-channel --target=http://localhost:3000

Now, if you push something to your repo, you should see something like this

Conclusion

We have successfully built our own Webhook payload delivery service.

I hope this was a fun little project and that maybe you learned something new.

SSE is awesome, and there are plenty of other things we can do with it.

I plan on writing a second part about using this to build a Telegram Bot.

Thanks for reading and feel free to ask any questions!

Appendices

Deploying the server

As I mentioned before, if you already have a preferred way of deploying your application, go for it.

I will show how I like to deploy my applications.

1. Hosting

We will be hosting our server at DigitalOcean, so make sure you have an account set up and that you are logged in.

First, we will create a new Droplet (A cloud server)

Choose the region that best fits your needs (probably the one closest to where you live)

Then, you can choose Ubuntu as the operating system, with the simplest machine available (A regular SSD: 1 GB / 1 CPU, 25 GB SSD Disk, 1000 GB transfer)

Select an SSH key as our authentication method and configure it

Finally, create the Droplet

After the Droplet is created, you will have access to its IP

Then, make sure everything is working by logging into the VM through SSH

ssh root@<your-ip-address>

Aaand, we are on!

2. Configuring Docker

For the actual deployment, we will need to "Dockerize" our project.

Since this is a simple Go application, we will only need a Dockerfile at the root of the project

# 1. build the server
FROM golang:1.23.1 AS builder
WORKDIR /app
COPY . .
ENV CGO_ENABLED=0
ENV GOOS=linux
RUN go build -o app ./cmd/web/

# 2. run the server
FROM scratch
# This is needed to include CA certificates in the `scratch` image, enabling your app to verify HTTPS requests, as `scratch` is completely empty by default.
# since golang:1.23.1 image is based on Debian or similar distributions, and it comes with CA certificates pre-installed, we can just copy from there
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
WORKDIR /app
COPY --from=builder /app/app .
EXPOSE 80
CMD ["./app", "-addr", ":80"]

3. Deploy with Kamal

For actually deploying the application, I really enjoy using Kamal, and with its newest version, it makes all the process a breeze.

Kamal offers zero-downtime deploys, rolling restarts, asset bridging, remote builds, accessory service management, and everything else you need to deploy and manage your web app in production with Docker. Originally built for Rails apps, Kamal will work with any type of web app that can be containerized.

First, make sure you have Kamal installed, you can achieve that by following their installation guide
In the end, you should be able to run Kamal through the terminal

Then, let's init Kamal into our application

kamal init

Which should give you something like this

Now, you should see a config/deploy.yml file and a .kamal folder

Now, you can give a name for your service and image, fill in your webserver IP address and provide the credentials for your image host

You should have something like this in the end

# Name of your application. Used to uniquely configure containers.
service: my-go-webhook-deliveries

# Name of the container image.
image: kuroski/my-go-webhook-deliveries

# Deploy to these servers.
servers:
  web:
    - 206.189.59.71

# You can configure a hostname later
# This will Enable SSL auto certification via Let's Encrypt and allow for multiple apps on a single web server.
#proxy:
#  ssl: true
#  host: app.example.com
  # Proxy connects to your container on port 80 by default.
  # app_port: 3000

# Credentials for your image host.
# By default, it uses Docker Hub, so, just insert your username there
# https://hub.docker.com/
registry:
  username: kuroski
  password:
    - KAMAL_REGISTRY_PASSWORD

# Configure builder setup.
builder:
  arch: amd64

The last piece of the puzzle is to add the KAMAL_REGISTRY_PASSWORD to your environment variables

Kamal manages environment variables through .kamal/secrets file.

By default, it reads secrets from your computer environment

KAMAL_REGISTRY_PASSWORD=$KAMAL_REGISTRY_PASSWORD

So, generate a new access token in DockerHub with full access permission

And then add it to your profile accordingly (~/.zshr, ~/.bashrc, ~/.config/fish/config.fish, etc...)

I prefer to store my secrets in a password manager like 1Password instead, and Kamal has a helper that reads from several different password managers
In case you have 1Password, you can store your credentials there and access it by having the following command in your .kamal/secrets file instead

SECRETS=$(kamal secrets fetch --adapter 1password --account my.1password.com --from <your-vault>/<name-of-the-vault-item> DOCKER_HUB_ACCESS_TOKEN) # DOCKER_HUB_ACCESS_TOKEN is just an alias, you can name an item the way you want

# then, you can use the "DOCKER_HUB_ACCESS_TOKEN" variable from your 1Password and store its value into the required "KAMAL_REGISTRY_PASSWORD" variable
KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract DOCKER_HUB_ACCESS_TOKEN $SECRETS)

And we need to include a "health check" endpoint, since the proxy will by default hit /up once every second until we hit the deploy timeout, with a 5-second timeout for each request.

// internal/server/routes.go

package server

import (
    "net/http"
)

func (srv *Server) routes() http.Handler {
    mux := http.NewServeMux()
     // we need this "/up" endpoint
    mux.HandleFunc("GET /up", srv.upHandler)
    mux.HandleFunc("GET /channel/{channel}", srv.subscribeToChannelHandler)
    mux.HandleFunc("POST /channel/{channel}", srv.forwardRequestToChannelHandler)
    return mux
}

// internal/server/handlers.go

package server

import (
    "encoding/json"
    "fmt"
    "github.com/kuroski/go-webhook-deliveries/internal/model"
    "github.com/tmaxmax/go-sse"
    "io"
    "net/http"
    "time"
)

func (srv *Server) upHandler(w http.ResponseWriter, r *http.Request) {
    if _, err := fmt.Fprintf(w, "Hello World!!"); err != nil {
        srv.serverError(w, r, err)
    }
}

// ...

Now, if everything is set up correctly, you need only to run kamal setup for the first run, and for future deployments, you can run kamal deploy

kamal setup # Setup all accessories, push the env, and deploy app to servers
kamal deploy # Deploy app to servers

Please make sure you have added the application under Git, otherwise Kamal won't work

When everything is done, you will be finally able to access the application, you can try accessing the health check endpoint http://<your-server-ip>/up

This should be enough for you to experiment with the application we made on this tutorial.

Extras

As a couple of extra points, you can

Add your own domain name and set up SSL

I enjoy using https://www.namecheap.com/ for that.

After you acquire your domain, you can configure it in your DigitalOcean server

And then update your Kamal config file

# ....

# You can configure a hostname later
# This will Enable SSL auto certification via Let's Encrypt and allow for multiple apps on a single web server.
#proxy:
ssl: true
  host: my-domain.com

# .....

And the reeeally cool thing is that configuring subdomains is a breeze, change the host attribute to your subdomain

ssl: true
  host: hello.my-domain.com

And then include it the A record at DigitalOcean

Kamal will also take care of SSL certificates and subdomains 🎉

Add some security goods

Great, we have the server, we have HTTPS, but there are plenty of things that can be made to optimise and secure the server.

For playing around, you might not need anything else, but if you want to give it a go, please check out this awesome repository

https://github.com/guillaumebriday/kamal-ansible-manager
This is an Ansible playbook to automatically optimise and secure your servers for Kamal, for Ubuntu only.

And the author made a great video explaining how things work.

General improvements in the dev environment

You can create a compose.dev.yml to provide some basic setup for your application

x-common_config: &common_config
  image: golang:1.23.1
  working_dir: /app
  networks:
    - app
  volumes:
    - .:/app

services:
  web:
    <<: *common_config
    ports:
      - "3000:3000"
    command: go run ./cmd/web
    develop:
      watch:
        - action: sync+restart
          path: ./cmd/web
          target: /app
        - action: sync+restart
          path: ./internal
          target: /app
    restart: on-failure
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/up"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s

  cli:
    <<: *common_config
    depends_on:
      - web
    develop:
      watch:
        - action: sync+restart
          path: ./cmd/cli
          target: /app
    command: go run ./cmd/cli/main.go --source="${DEV_CLI_SOURCE_URL}" --target="${DEV_CLI_TARGET_URL}"
    restart: on-failure:3

networks:
  app:

volumes:
  app:

Then creating a Makefile

include .env

# ==================================================================================== #
# CONSTANTS
# ==================================================================================== #
IMAGE_NAME=my-go-webhook-deliveries
CONTAINER_NAME=my-go-webhook-deliveries

# ==================================================================================== #
# HELPERS
# ==================================================================================== #
## Prints usage information for each target
.PHONY: help
help:
    @echo 'Usage:'
    @sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' | sed -e 's/^/ /'
.PHONY: confirm
confirm:
    @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ]

# ==================================================================================== #
# DEVELOPMENT
# ==================================================================================== #
## Web Application: Runs the web application without docker
.PHONY: web-run
web-run:
    go run ./cmd/web

## Client: Runs the client CLI
.PHONY: cli-run
cli-run:
    @source_arg=$${source:-${DEV_CLI_SOURCE_URL}}; \
    target_arg=$${target:-${DEV_CLI_TARGET_URL}}; \
    go run ./cmd/cli --source=$$source_arg --target=$$target_arg

## Dev: Runs dev environment with docker compose
.PHONY: dev
dev:
    docker compose -f compose.dev.yml up
    @echo 'Your server is running on localhost:3000'


# ==================================================================================== #
# QUALITY CONTROL
# ==================================================================================== #
## Tidy: Formats .go files and tidies dependencies
.PHONY: tidy
tidy:
    @echo 'Formatting .go files...'
    go fmt ./...
    @echo 'Tidying module dependencies...'
    go mod tidy

## Audit: Runs quality control checks
.PHONY: audit
audit:
    @echo 'Checking module dependencies'
    go mod tidy -diff
    go mod verify
    @echo 'Vetting code...'
    go vet ./...
    staticcheck ./...
    @echo 'Running tests...'
    go test -race -vet=off ./...


# ==================================================================================== #
# PRODUCTION
# ==================================================================================== #
## Deploy the application (skipping Kamal hooks)
.PHONY: deploy
deploy:
    @echo 'Deploying the app...'
    kamal deploy -H
    @echo 'Tidying module dependencies...'
    go mod tidy

This way you can just make dev and have everything up and running

DEV Community