Building GitHub Apps with Golang

If you're using GitHub as your version control system of choice then GitHub Apps can be incredibly useful for many tasks including building CI/CD, managing repositories, querying statistical data and much more. In this article we will walk through the process of building such an app in Go including setting up the GitHub integration, authenticating with GitHub, listening to webhooks, querying GitHub API and more.

TL;DR: All the code used in this article is available at https://github.com/MartinHeinz/go-github-app.

Choosing Integration Type

Before we jump into building the app, we first need to decide which type of integration we want to use. GitHub provides 3 options - Personal Access Tokens, GitHub Apps and OAuth Apps. Each of these 3 have their pros and cons, so here are some basic things to consider:

• Personal Access Token is the simplest form of authentication and is suitable if you only need to authenticate with GitHub as yourself. If you need to act on behalf of other users, then this won't be good enough
• GitHub Apps are the preferred way of developing GitHub integrations. They can be installed by individual users as well as whole organizations. They can listen to events from GitHub via webhooks as well as access the API when needed. They're quite powerful, but even if you request all the permissions available, you won't be able to use them to perform all the actions that a user can.
• OAuth Apps use OAuth2 to authenticate with GitHub on behalf of user. This means that they can perform any action that user can. This might seem like the best option, but the permissions don't provide the same granularity as GitHub Apps, and it's also more difficult to set up because of OAuth.

If you're not sure what to choose, then you can also take a look at diagram in docs which might help you decide. In this article we will use GitHub App as it's very versatile integration and best option for most use cases.

Setting Up

Before we start writing any code, we need to create and configure the GitHub App integration:

1. As a prerequisite, we need a tunnel which we will use to deliver GitHub webhooks from internet to our locally running application. You will need to install localtunnel tool with npm install -g localtunnel and start forwarding to your localhost using lt --port 8080.

2. Next we need to go to https://github.com/settings/apps/new to configure the integration. Fill the fields as follows:

   • Homepage URL: Your localtunnel URL
   • Webhook URL: https://<LOCALTUNNEL_URL>/api/v1/github/payload
   • Webhook secret: any secret you want (and save it)
   • Repository Permissions: Contents, Metadata (Read-only)
   • Subscribe to events: Push, Release

3. After creating the app, you will be presented with the settings page of the integration. Take note of App ID, generate a private key and download it.

4. Next you will also need to install the app to use it with your GitHub account. Go to Install App tab and install it into your account.

5. We also need installation ID, which we can find by going to Advanced tab and clicking on latest delivery in the list, take a note of installation ID from request payload, it should be located in { "installation": { "id": <...>} }.

If you've got lost somewhere along the way, refer to the guide GitHub docs which shows where you can find each of the values.

With that done, we have the integration configured and all the important values saved. Before we start receiving events and making API requests we need to get the Go server up and running, so let's start coding!

Building the App

To build the Go application, we will use the template I prepared in https://github.com/MartinHeinz/go-github-app. This application is ready to be used as GitHub app and all that's missing in it, are a couple of variables which we saved during setup in previous section. The repository contains convenience script which you can use to populate all the values:

git clone [e-mail]:MartinHeinz/go-github-app.git && cd go-github-app
./configure_project.sh \
    APP_ID="54321" \
    INSTALLATION_ID="987654321" \
    WEBHOOK_SECRET="verysecret" \
    KEY_PATH="./github_key.pem" \
    REGISTRY="ghcr.io/<GITHUB_USERNAME>/go-github-app"

The following sections will walk you through the code but if you're inpatient, then the app is good to go. You can use make build to build a binary of the application or make container to create a containerized version of it.

First part of the code we need to tackle is authentication. It's done using ghinstallation package as follows:

func InitGitHubClient() {
    tr := http.DefaultTransport
    itr, err := ghinstallation.NewKeyFromFile(tr, 12345, 123456789, "/config/github-app.pem")

    if err != nil {
        log.Fatal(err)
    }

    config.Config.GitHubClient = github.NewClient(&http.Client{Transport: itr})
}

This function, which is invoked from main.go during Gin server start-up, takes App ID, Installation ID and private key to create a GitHub client which is then stored in global config in config.Config.GitHubClient. We will use this client to talk to the GitHub API later.

Along with the GitHub client, we also need to set up server routes so that we can receive payloads:

func main() {
    // ...
    v1 := r.Group("/api/v1")
    {
        v1.POST("/github/payload", webhooks.ConsumeEvent)
        v1.GET("/github/pullrequests/:owner/:repo", apis.GetPullRequests)
        v1.GET("/github/pullrequests/:owner/:repo/:page", apis.GetPullRequestsPaginated)
    }
    utils.InitGitHubClient()
    r.Run(fmt.Sprintf(":%v", config.Config.ServerPort))
}

First of these is the payload path at http://.../api/v1/github/payload which we used during GitHub integration setup. This path is associated with webhooks.ConsumeEvent function which will receive all the events from GitHub.

For security reasons, the first thing the webhooks.ConsumeEvent function does is verify request signature to make sure that GitHub is really the service that generated the event:

func VerifySignature(payload []byte, signature string) bool {
    key := hmac.New(sha256.New, []byte(config.Config.GitHubWebhookSecret))
    key.Write([]byte(string(payload)))
    computedSignature := "sha256=" + hex.EncodeToString(key.Sum(nil))
    log.Printf("computed signature: %s", computedSignature)

    return computedSignature == signature
}

func ConsumeEvent(c *gin.Context) {
    payload, _ := ioutil.ReadAll(c.Request.Body)

    if !VerifySignature(payload, c.GetHeader("X-Hub-Signature-256")) {
        c.AbortWithStatus(http.StatusUnauthorized)
        log.Println("signatures don't match")
    }
    // ...

It performs the verification by computing a HMAC digest of payload using webhook secret as a key, which is then compared with the value in X-Hub-Signature-256 header of a request. If the signatures match then we can proceed to consuming the individual events:

func ConsumeEvent(c *gin.Context) {
    // ...
    event := c.GetHeader("X-GitHub-Event")

    for _, e := range Events {
        if string(e) == event {
            log.Printf("consuming event: %s", e)
            var p EventPayload
            json.Unmarshal(payload, &p)
            if err := Consumers[string(e)](p); err != nil {
                log.Printf("couldn't consume event %s, error: %+v", string(e), err)
                // We're responding to GitHub API, we really just want to say "OK" or "not OK"
                c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"reason": err})
            }
            log.Printf("consumed event: %s", e)
            c.AbortWithStatus(http.StatusNoContent)
            return
        }
    }
    log.Printf("Unsupported event: %s", event)
    c.AbortWithStatusJSON(http.StatusNotImplemented, gin.H{"reason": "Unsupported event: " + event})
}

In the above snippet we extract the event type from X-GitHub-Event header and iterate through a list of events that our app supports. In this case those are:

const (
    Install     Event = "installation"
    Ping        Event = "ping"
    Push        Event = "push"
    PullRequest Event = "pull_request"
)

var Events = []Event{
    Install,
    Ping,
    Push,
    PullRequest,
}

If the event name matches one of the options we proceed with loading the JSON payload into a EventPayload struct, which is defined in cmd/app/webhook/models.go. It's just a struct generated using https://mholt.github.io/json-to-go/ with unnecessary fields stripped.

That payload is then sent to function that handles the respective event type, which is one of the following:

var Consumers = map[string]func(EventPayload) error{
    string(Install):     consumeInstallEvent,
    string(Ping):        consumePingEvent,
    string(Push):        consumePushEvent,
    string(PullRequest): consumePullRequestEvent,
}

For example for push event one can do something like this:

func consumePushEvent(payload EventPayload) error {
    // Process event ...
    // Insert data into database ...
    log.Printf("Received push from %s, by user %s, on branch %s",
        payload.Repository.FullName,
        payload.Pusher.Name,
        payload.Ref)

    // Enumerating commits
    var commits []string
    for _, commit := range payload.Commits {
        commits = append(commits, commit.ID)
    }
    log.Printf("Pushed commits: %v", commits)

    return nil
}

That being in this case - checking the receiving repository and branch and enumerating the commits contained in this single push. This is the place where you could for example insert the data into database or send some notification regarding the event.

Now we have the code ready, but how do we test it? To do so, we will use the tunnel which you already should have running, assuming you followed the steps in previous sections.

Additionally, we also need to spin up the server, you can do that by running make container to build the containerized application, followed by make run which will start the container that listens on port 8080.

Now you can simply push to one of your repositories and you should see a similar output in the server logs:

[GIN] 2022/01/02 - 14:44:10 | 204 |     696.813µs |   123.82.234.90 | POST     "/api/v1/github/payload"
2022/01/02 14:44:10 Received push from MartinHeinz/some-repo, by user MartinHeinz, on branch refs/heads/master
2022/01/02 14:44:10 Pushed commits: [9024da76ec611e60a8dc833eaa6bca7b005bb029]
2022/01/02 14:44:10 consumed event: push

To avoid having to push dummy changes to repositories all the time, you can redeliver payloads from Advanced tab in your GitHub App configuration. On this tab you will find a list of previous requests, just choose one and hit the Redeliver button.

Making API Calls

GitHub apps are centered around webhooks to which you can subscribe and listen to, but you can also use any of the GitHub REST/GraphQL API endpoints assuming you requested the necessary permissions. Using API rather than push events is useful - for example - when creating files, analyzing bulk data or querying data which cannot be received from webhooks.

For demonstration of how to do so, we will retrieve pull requests of specified repository:

func GetPullRequests(c *gin.Context) {
    owner := c.Param("owner")
    repo := c.Param("repo")
    if pullRequests, resp, err := config.Config.GitHubClient.PullRequests.List(
        c, owner, repo, &github.PullRequestListOptions{
        State: "open",
    }); err != nil {
        log.Println(err)
        c.AbortWithStatus(resp.StatusCode)
    } else {
        var pullRequestTitles []string
        for _, pr := range pullRequests {
            pullRequestTitles = append(pullRequestTitles, *pr.Title)
        }
        c.JSON(http.StatusOK, gin.H{
            "pull_requests": pullRequestTitles,
        })
    }
}

This function takes 2 arguments - owner and repo - which get passed to PullRequests.List(...) function of GitHub client instance. Along with that, we also provide PullRequestListOptions struct to specify that we're only interested in pull requests with state set to open. We then iterate over returned PRs and accumulate all their titles which we return in response.

The above function resides on .../api/v1/github/pullrequests/:owner/:repo path as specified in main.go so we can query it like so:

curl http://localhost:8080/api/v1/github/pullrequests/octocat/hello-world | jq .

It might not be ideal to query API as shown above in situations where we expect a lot of data to be returned. In those cases we can utilize paging to avoid hitting rate limits. A function called GetPullRequestsPaginated that performs the same task as GetPullRequests with addition of page argument for specifying page size can be found in cmd/app/apis/github.go.

Writing Tests

So far we've been testing the app with localtunnel, which is nice for quick ad-hoc tests against live API, but it doesn't replace proper unit tests. To write unit tests for this app, we need to mock-out the API to avoid being dependent on the external service. To do so, we can use go-github-mock:

func TestGithubGetPullRequests(t *testing.T) {
    expectedTitles := []string{ "PR number one", "PR number three" }
    closedPullRequestTitle := "PR number two"
    mockedHTTPClient := mock.NewMockedHTTPClient(
        mock.WithRequestMatch(
            mock.GetReposPullsByOwnerByRepo,
            []github.PullRequest{
                {State: github.String("open"), Title: &expectedTitles[0]},
                {State: github.String("closed"), Title: &closedPullRequestTitle},
                {State: github.String("open"), Title: &expectedTitles[1]},
            },
        ),
    )
    client := github.NewClient(mockedHTTPClient)
    config.Config.GitHubClient = client

    gin.SetMode(gin.TestMode)
    res := httptest.NewRecorder()
    ctx, _ := gin.CreateTestContext(res)
    ctx.Params = []gin.Param{
        {Key: "owner", Value: "octocat"},
        {Key: "repo", Value: "hello-world"},
    }

    GetPullRequests(ctx)
    body, _ := ioutil.ReadAll(res.Body)

    assert.Equal(t, 200, res.Code)
    assert.Contains(t, string(body), expectedTitles[0])
    assert.NotContains(t, string(body), closedPullRequestTitle)
    assert.Contains(t, string(body), expectedTitles[1])
}

This test starts by defining mock client which will be used in place of normal GitHub client. We give it list of pull request which will be returned when PullRequests.List is called. We then create test context with arguments that we want to pass to the function under test, and we invoke the function. Finally, we read the response body and assert that only PRs with open state were returned.

For more tests, see the full source code which includes examples of tests for pagination as well as handling of errors coming from GitHub API.

When it comes to testing our webhook methods, we don't need to use a mock client, because we're dealing with basic API requests. Example of such tests including generic API testing setup can be found in cmd/app/webhooks/github_test.go.

Conclusion

In this article I tried to give you a quick tour of both GitHub apps, as well as the GitHub repository containing the sample Go GitHub project. In both cases, I didn't cover everything, the Go client package has much more to offer and to see all the actions you can perform with it, I recommend skimming through the docs index as well as looking at the source code itself where GitHub API links are listed along each function. For example, like the earlier shown PullRequests.List here.

As for the repository, there are couple more things you might want to take a look at, including Makefile targets, CI/CD or additional tests. If you have any feedback or suggestions, feel free to create an issue or just star it if it was helpful to you. 🙂

Comments

[M Zhuo]

Thank you for this good article and example!

[Harshad Saykhedkar]

Well written article, explained all the steps in detail. The code is quite structured and readable too. All in all, good stuff!