Andy Haskell for Salesforce Engineering

Posted on Jul 23, 2021

Test your Go web apps with httptest

#go #testing #codenewbie #webdev

In part 1 of this series, we looked at the basics of writing tests in Go with the testing.T type, and in part 2, we looked at how with just the testing.T type, you can organize your tests with its Run, Skip, and Cleanup methods. Even with a good grasp of just testing.T, you're ready to write professional test coverage in your Go codebase.

But testing.T and isn't all that the Go standard library provides for writing Go tests! One of the most popular uses of Go is building the server code for web apps, using its detailed net/http package. So the Go Team provided us with a sweet additional package for testing web apps in addition to the main testing package: net/http/httptest!

In this tutorial we'll look at:

📋 how to test an HTTP handler with httptest
💻 how to use httptest to test against a real server

This tutorial is for you if you're interested in Go web development or do webdev already, you're familiar with the basics of Go testing and structs, and you have some familiarity with the HTTP protocol's concepts of requests, responses, headers. If you've used the net/http package, that will help you follow along, but if you're new to net/http, this tutorial does have an overview of some Go web app concepts.

📶 A quick overview of Go HTTP handlers

If you already are familiar with Go HTTP handlers, feel free to read the code sample and then skip ahead to the next section. If you're new to Go web development or just want a recap, read on!

Let's start with a recap of one of its core concepts: when you get an HTTP request, you process it using a handler.

Handlers look something like the handleSlothfulMessage function in this code sample (if you're following along, save this to a file titled server.go):

package main

import (
    "net/http"
)

func handleSlothfulMessage(w http.ResponseWriter, r *http.Request) {
    w.Header().Add("Content-Type", "application/json")
    w.Write([]byte(`{"message": "Stay slothful!"}`))
}

func appRouter() http.Handler {
    rt := http.NewServeMux()
    rt.HandleFunc("/sloth", handleSlothfulMessage)
    return rt
}

func main() { http.ListenAndServe(":1123", appRouter()) }

All Go HTTP handlers take in an implementation of the ResponseWriter interface, and a Request struct. Those objects have the following responsibilities:

The Request contains the data of the HTTP request that the HTTP client (ex a browser or cURL), sends to your web server, like:
- The URL that was requested, like the path and query parameters.
- Request headers, for example a User-Agent header saying whether the request comes from Firefox, Chrome, a command-line client, or a punched card someone delivered to a card reader.
- The request body for POST and PUT requests.
The ResponseWriter is in charge of formulating the HTTP response, so it handles things like:
- Writing status codes, like 200 for a successful request, or the familiar "404 file not found" websites make corny webpages for.
- Writing Response headers, such as the Content-Type to say what format our response is in, like HTML or JSON.
- Writing the response body, which can be any response format, like an HTML webpage, a JSON object, or picture of a sloth 🌺!

In our handleSlothfulMessage function, we first add a header to indicate that our response is JSON with the line:

w.Header().Add("Content-Type", "application/json")

And then we write out the bytes of a JSON message that has a header saying "Stay slothful!" with the line:

w.Write([]byte(`{"message": "Stay slothful!"}`))

Note that because we didn't call w.WriteHeader to explicitly select a status code for our HTTP response, the response's code will be 200/OK. If we had some kind of error scenario we'd need to handle in one of our web app's handlers, for example issues talking to a database, then in the handler function we might have code like this to give a 500/internal server error response:

if errorScenarioOccurs {
    w.WriteHeader(http.StatusInternalServerError)
    w.Write([]byte(`{"error_message": "description of the error"}`))
    return
}

// rest of HTTP handler

In the appRouter function, we make a router so requests to our /sloth endpoint are handled with the handleSlothfulMessage function:

func appRouter() http.Handler {
    rt := http.NewServeMux()
    rt.HandleFunc("/sloth", handleSlothfulMessage)
    return rt
}

Finally, in the main function, we start an HTTP server on port 1123 with http.ListenAndServe, causing all requests to localhost:1123 to be handled by the HTTP router we made in appRouter.

Now if you start this Go program and then go to localhost:1123/sloth in your browser or send a request to it via a client like cURL or Postman, you can see that we got back a simple JSON object!

📋 Testing your handler with httptest

As you can see, you can start using your HTTP handler in a real web server without a lot of code. When you're running a net/http server with http.ListenAndServe, Go does the work behind the scenes for you of making http.Request and ResponseWriter objects when a request comes in from a client like your browser.

But that does raise the question, where do we get a ResponseWriter and Request in our Go code when we're testing our HTTP handlers inside go test?

Luckily, the standard library has convenient code for testing that in the net/http/httptest package to facilitate writing test coverage for that, with functions and types like:

The NewRequest function for making the *http.Request.
a ResponseRecorder type that both implements the http.ResponseWriter interface and lets you replay the HTTP response.
a Server type for testing Go code that sends HTTP requests by setting up a real HTTP server to send them to.

To see this httptest in action, let's see how we would test handleSlothfulMessage. If you're following along, save this code to server_test.go:

package main

import (
    "net/http"
    "net/http/httptest"
    "strings"
    "testing"
)

func TestHandleSlothfulMessage(t *testing.T) {
    wr := httptest.NewRecorder()
    req := httptest.NewRequest(http.MethodGet, "/sloth", nil)

    handleSlothfulMessage(wr, req)
    if wr.Code != http.StatusOK {
        t.Errorf("got HTTP status code %d, expected 200", wr.Code)
    }

    if !strings.Contains(wr.Body.String(), "Stay slothful!") {
        t.Errorf(
            `response body "%s" does not contain "Stay slothful!"`,
            wr.Body.String(),
        )
    }
}

Run go test -v, and you should see a passing test!

Let's take a look at what happened, and how we're using httptest in our code:

First, there's making the ResponseWriter and Request:

    wr := httptest.NewRecorder()
    req := httptest.NewRequest(http.MethodGet, "/sloth", nil)

We make our net/http ResponseWriter implementation with NewRecorder, and a Request object pointed at our /sloth endpoint using NewRequest.

Then, we run our HTTP handler:

handleSlothfulMessage(wr, req)

Remember that Go HTTP handler functions are just regular old Go functions that happen to take in specialized net/http objects. That means we can run a handler without any actual HTTP server if we have a ResponseWriter and Request to pass into it. So we run our handler by passing wr and req into a call to our handleSlothfulMessage function. Or if we wanted to test our web app's entire router rather than just one endpoint, we could even run appRouter().ServeHTTP(wr, req)!

Then, in the next piece of code, we check out the results of running handleSlothfulMessage:

    if wr.Code != http.StatusOK {
        t.Errorf("got HTTP status code %d, expected 200", wr.Code)
    }

An httptest ResponseRecorder implements the ResponseWriter interface, but that's not all it gives us! It also has struct fields we can use for examining the response we get back from our HTTP request. One of them is Code; we expect our response to be a 200, so we have an assertion comparing our status code to http.StatusOK.

Additionally, a ResponseRecorder makes it easy to look at the body of our response. It gives us a bytes.Buffer field titled Body that recorded the bytes of the response body. So we can test that our HTTP response contains the string "Stay slothful!", having our test fail if it does not.

    if !strings.Contains(wr.Body.String(), "Stay slothful!") {
        t.Errorf(
            `response body "%s" does not contain "Stay slothful!"`,
            wr.Body.String(),
        )
    }

By the way, this technique also works with POST requests. If we had an endpoint that took in a POST request with an encoded JSON body, then sending the request in the test for that endpoint would look something like this:

var b bytes.Buffer
err := json.NewEncoder(b).Encode(objectToSerializeToJSON)
if err != nil {
    t.Fatal(err)
}

wr := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodPost, "/post-endpoint", b)
handlePostEndpointRequest(wr, req)

First we set up a bytes.Buffer to use as our POST request's body. This is because a net/http Request's body needs to be an implementation of the io.Reader interface. bytes.Buffer conveniently has the Read method, so it implements that interface. We then use json.NewEncoder(b).Encode to convert a Go struct into bytes of JSON that get stored in the buffer.

We make our POST request by passing MethodPost, rather than MethodGet, into httptest.NewRequest. Our bytes.Buffer is passed in as the last argument to NewRequest as the request body. Finally, just like before, we call our HTTP request handler using our ResponseRecorder and Request.

💻 The httptest server

Not only does httptest provide us a way to test our handlers with requests and responses, it even provides ways to test your code with a real HTTP server!

A couple scenarios where this is useful are:

When you're doing functional tests or integration tests for a web app. For example, testing that communication goes as expected between different microservices.
If you are implementing clients to other web servers, you can define an httptest server giving back a response in order to test that your client can handle both sending the correct request and processing the server's response correctly.

Let's try out the latter of these scenarios, by making a client struct to send an HTTP request to our /sloth endpoint, and deserialize the response into a struct.

First, import fmt and encoding/json (and net/http if you're putting the client code in its own file) and then write this code for the client. If you're newer to JSON deserialization, no worries if the code doesn't click 100% for you. The main things you need to know are:

The client we're making has a GetSlothfulMessage that sends an HTTP request to the /sloth of its baseURL.
Using Go's awesome encoding/json package, the HTTP response body is converted to a SlothfulMessage struct, which is returned if the request and JSON deserialization are successful. We are using json.NewDecoder(res.Body).Decode for reading the response body into our SlothfulMessage struct.
If we get a non-200 HTTP status code sending the request, or there's a problem deserializing the JSON response, then GetSlothfulMessage instead returns an error.

type Client struct {
    httpClient *http.Client
    baseURL string
}

type SlothfulMessage struct {
    Message string `json:"message"`
}

func NewClient(httpClient *http.Client, baseURL string) Client {
    return Client{
        httpClient: httpClient,
        baseURL:    baseURL,
    }
}

func (c *Client) GetSlothfulMessage() (*SlothfulMessage, error) {
    res, err := c.httpClient.Get(c.baseURL + "/sloths")
    if err != nil {
        return nil, err
    }

    if res.StatusCode != http.StatusOK {
        return nil, fmt.Errorf(
            "got status code %d", res.StatusCode,
        )
    }

    var m SlothfulMessage
    if err := json.NewDecoder(res.Body).Decode(&m); err != nil {
        return nil, err
    }
    return &m, nil
}

We've got our client, so let's see how we can test it with an httptest.Server:

func TestGetSlothfulMessage(t *testing.T) {
    router := http.NewServeMux()
    router.HandleFunc("/sloth", handleSlothfulMessage)

    svr := httptest.NewServer(router)
    defer svr.Close()

    c := NewClient(http.DefaultClient, svr.URL)
    m, err := c.GetSlothfulMessage()
    if err != nil {
        t.Fatalf("error in GetSlothfulMessage: %v", err)
    }
    if m.Message != "Stay slothful!" {
        t.Errorf(
            `message %s should contain string "Sloth"`,
            m.Message,
        )
    }
}

Here's what happens:

First, we start our server:

    svr := httptest.NewServer(appRouter())
    defer svr.Close()

We pass in our web app's HTTP router as a request handler for the server. When we run NewServer, a server is set up to run on your localhost, on a randomized port. In fact, if you had your test run time.Sleep to pause for a while, you could actually go to that server in your own browser!

Now that we've got our server, we set up our client and have it test an HTTP roundtrip to our /sloth endpoint:

    c := NewClient(http.DefaultClient, svr.URL)
    m, err := c.GetSlothfulMessage()

The base URL we give to the Client, is the URL of the server, which is the randomized port I mentioned earlier. So a request might go out to somewhere like "localhost:1123/sloths", or "localhost:5813/sloths". It all depends on which port httptest.NewServer picks!

Finally, we check that we didn't get an error, and that the response is what we expected. If we run go test -v, we'll get:

=== RUN   TestHandleSlothfulMessage
-------- PASS: TestHandleSlothfulMessage (0.00s)
=== RUN   TestGetSlothfulMessage
    webapp_test.go:37: error in GetSlothfulMessage: got status code 404
-------- FAIL: TestGetSlothfulMessage (0.00s)

A failing test, because we got a 404 response, not the 200 we expected. So that means there's a bug in our client.

The part of GetSlothfulMessage that was for sending our HTTP request was:

func (c *Client) GetSlothfulMessage() (*SlothfulMessage, error) {
    res, err := c.httpClient.Get(c.baseURL + "/sloths")
    if err != nil {
        return nil, err
    }

As you can see, we're sending the request to c.baseURL + "/sloths". We wanted to send it to /sloth, not /sloths. So fix that code, run go test -v, and now...

=== RUN   TestHandleSlothfulMessage
-------- PASS: TestHandleSlothfulMessage (0.00s)
=== RUN   TestGetSlothfulMessage
-------- PASS: TestGetSlothfulMessage (0.00s)

Your test should pass!

As you can see, with the httptest package's ResponseRecorder and Server objects, we've got the ability to take the concepts we were already working with for writing tests using the testing package, and then start using functionality to test both receiving and sending HTTP requests. Definitely a must-know package in a Go web developer's toolbelt!