Improving Request, Validation, and Response Handling in Go Microservices

This guide explains how I streamlined the handling of requests, validations, and responses in my Go microservices, aiming for simplicity, reusability, and a more maintainable codebase.

Introduction

I've been working with microservices in Go for quite some time, and I always appreciate the clarity and simplicity that this language offers. One of the things I love most about Go is that nothing happens behind the scenes; the code is always transparent and predictable.

However, some parts of development can be quite tedious, especially when it comes to validating and standardizing responses in API endpoints. I have tried many different approaches to tackle this, but recently, while writing my Go course, I came up with a rather unexpected idea. This idea added a touch of “magic” to my handlers, and, to my surprise, I liked it. With this solution, I was able to centralize all the logic for validation, decoding, and parameter parsing of requests, as well as unify the encoding and responses for the APIs. In the end, I found a balance between maintaining code clarity and reducing repetitive implementations.

The Problem

When developing Go microservices, one common task is handling incoming HTTP requests efficiently. This process typically involves parsing request bodies, extracting parameters, validating the data, and sending back consistent responses. Let me illustrate the problem with an example:

package main

import (
 "encoding/json"
 "github.com/go-chi/chi/v5"
 "github.com/go-chi/chi/v5/middleware"
 "github.com/go-playground/validator/v10"
 "log"
 "net/http"
)

type SampleRequest struct {
 Name string `json:"name" validate:"required,min=3"`
 Age  int    `json:"age" validate:"required,min=1"`
}

var validate = validator.New()

type ValidationErrors struct {
 Errors map[string][]string `json:"errors"`
}

func main() {
 r := chi.NewRouter()
 r.Use(middleware.Logger)
 r.Use(middleware.Recoverer)

 r.Post("/submit/{name}", func(w http.ResponseWriter, r *http.Request) {
  sampleReq := &SampleRequest{}

  // Set the path parameter
  name := chi.URLParam(r, "name")
  if name == "" {
   w.WriteHeader(http.StatusBadRequest)
   json.NewEncoder(w).Encode(map[string]interface{}{
    "code":    http.StatusBadRequest,
    "message": "name is required",
   })
   return
  }
  sampleReq.Name = name

  // Parse and decode the JSON body
  if err := json.NewDecoder(r.Body).Decode(sampleReq); err != nil {
   w.WriteHeader(http.StatusBadRequest)
   json.NewEncoder(w).Encode(map[string]interface{}{
    "code":    http.StatusBadRequest,
    "message": "Invalid JSON format",
   })
   return
  }

  // Validate the request
  if err := validate.Struct(sampleReq); err != nil {
   validationErrors := make(map[string][]string)
   for _, err := range err.(validator.ValidationErrors) {
    fieldName := err.Field()
    validationErrors[fieldName] = append(validationErrors[fieldName], err.Tag())
   }
   w.WriteHeader(http.StatusBadRequest)
   json.NewEncoder(w).Encode(map[string]interface{}{
    "code":    http.StatusBadRequest,
    "message": "Validation error",
    "body":    ValidationErrors{Errors: validationErrors},
   })
   return
  }

  // Send success response
  w.WriteHeader(http.StatusOK)
  json.NewEncoder(w).Encode(map[string]interface{}{
   "code":    http.StatusOK,
   "message": "Request received successfully",
   "body":    sampleReq,
  })
 })

 log.Println("Starting server on :8080")
 http.ListenAndServe(":8080", r)
}

Let me explain the code above, focusing at the handler part where we manually handle:

• Handles path parameters: Verify if the required path parameters exists and handles it.
• Decoding the request body: Ensuring that the incoming JSON is parsed correctly.
• Validation: Using the validator package to check if the request fields meet the requirement criteria.
• Error handling: Responding to the client with appropriate error messages when validation fails or JSON is malformed.
• Consistent responses: Manually building a response structure.

While the code is functional, it involves a significant amount of boilerplate logic that must be repeated for each new endpoint, making it harder to maintain and prone to inconsistencies.

So, how can we improve this?

Breaking down the code

To address this issues and improve the code maintainability, I decided to split the logic into three distinct layers: Request, Response, and Validation. This approach encapsulates the logic for each part, making it reusable and easier to test independently.

Request Layer

The Request layer is responsible for parsing and extracting data from the incoming HTTP requests. By isolating this logic, we can standardize how data is processed and ensure that all parsing is handled uniformly.

Validation Layer

The Validation layer focuses solely on validating the parsed data according to predefined rules. This keeps validation logic separate from request handling, making it more maintainable and reusable across different endpoints.

Response Layer

The Response layer handler the construction and formatting of responses. By centralizing response logic, we can ensure that all API responses follow a consistent structure, simplifying debugging and improving client interactions.

So… Although splitting the code into layers offers benefits like reusability, testability, and maintainability, it comes with some trade-offs. Increased complexity can make the project structure harder for new developers to grasp, and for simple endpoints, using separate layers might feel excessive, potentially leading to over-engineering. Understanding these pros and cons helps in deciding when to apply this pattern effectively.

At the end of the day, is always about what is bothering you most. Right? So, now lets put some hand in our old code and start to implement the layers mentioned above.

Refactoring the code into layers

Step 1: Creating the Request Layer

First, we refactor the code to encapsulate request parsing into a dedicated function or module. This layer focuses solely on reading and parsing the request body, ensuring that it is decoupled from other responsibilities in the handler.

Create a new file httpsuite/request.go:

package httpsuite

import (
 "encoding/json"
 "errors"
 "github.com/go-chi/chi/v5"
 "net/http"
 "reflect"
)

// RequestParamSetter defines the interface used to set the parameters to the HTTP request object by the request parser.
// Implementing this interface allows custom handling of URL parameters.
type RequestParamSetter interface {
 // SetParam assigns a value to a specified field in the request struct.
 // The fieldName parameter is the name of the field, and value is the value to set.
 SetParam(fieldName, value string) error
}

// ParseRequest parses the incoming HTTP request into a specified struct type, handling JSON decoding and URL parameters.
// It validates the parsed request and returns it along with any potential errors.
// The pathParams variadic argument allows specifying URL parameters to be extracted.
// If an error occurs during parsing, validation, or parameter setting, it responds with an appropriate HTTP status.
func ParseRequest[T RequestParamSetter](w http.ResponseWriter, r *http.Request, pathParams ...string) (T, error) {
 var request T
 var empty T

 defer func() {
  _ = r.Body.Close()
 }()

 if r.Body != http.NoBody {
  if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
   SendResponse[any](w, "Invalid JSON format", http.StatusBadRequest, nil)
   return empty, err
  }
 }

 // If body wasn't parsed request may be nil and cause problems ahead
 if isRequestNil(request) {
  request = reflect.New(reflect.TypeOf(request).Elem()).Interface().(T)
 }

 // Parse URL parameters
 for _, key := range pathParams {
  value := chi.URLParam(r, key)
  if value == "" {
   SendResponse[any](w, "Parameter "+key+" not found in request", http.StatusBadRequest, nil)
   return empty, errors.New("missing parameter: " + key)
  }

  if err := request.SetParam(key, value); err != nil {
   SendResponse[any](w, "Failed to set field "+key, http.StatusInternalServerError, nil)
   return empty, err
  }
 }

 // Validate the combined request struct
 if validationErr := IsRequestValid(request); validationErr != nil {
  SendResponse[ValidationErrors](w, "Validation error", http.StatusBadRequest, validationErr)
  return empty, errors.New("validation error")
 }

 return request, nil
}

func isRequestNil(i interface{}) bool {
 return i == nil || (reflect.ValueOf(i).Kind() == reflect.Ptr && reflect.ValueOf(i).IsNil())
}

Note: At this point, I had to use reflection. Probably I'm way to stupid to find a better wait do do it. 😼

Of course, that we can also test this, create the test file httpsuite/request_test.go:

package httpsuite

import (
 "bytes"
 "context"
 "encoding/json"
 "errors"
 "fmt"
 "github.com/go-chi/chi/v5"
 "github.com/stretchr/testify/assert"
 "log"
 "net/http"
 "net/http/httptest"
 "strconv"
 "strings"
 "testing"
)

// TestRequest includes custom type annotation for UUID
type TestRequest struct {
 ID   int    `json:"id" validate:"required"`
 Name string `json:"name" validate:"required"`
}

func (r *TestRequest) SetParam(fieldName, value string) error {
 switch strings.ToLower(fieldName) {
 case "id":
  id, err := strconv.Atoi(value)
  if err != nil {
   return errors.New("invalid id")
  }
  r.ID = id
 default:
  log.Printf("Parameter %s cannot be set", fieldName)
 }

 return nil
}

func Test_ParseRequest(t *testing.T) {
 testSetURLParam := func(r *http.Request, fieldName, value string) *http.Request {
  ctx := chi.NewRouteContext()
  ctx.URLParams.Add(fieldName, value)
  return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, ctx))
 }

 type args struct {
  w          http.ResponseWriter
  r          *http.Request
  pathParams []string
 }
 type testCase[T any] struct {
  name    string
  args    args
  want    *TestRequest
  wantErr assert.ErrorAssertionFunc
 }
 tests := []testCase[TestRequest]{
  {
   name: "Successful Request",
   args: args{
    w: httptest.NewRecorder(),
    r: func() *http.Request {
     body, _ := json.Marshal(TestRequest{Name: "Test"})
     req := httptest.NewRequest("POST", "/test/123", bytes.NewBuffer(body))
     req = testSetURLParam(req, "ID", "123")
     req.Header.Set("Content-Type", "application/json")
     return req
    }(),
    pathParams: []string{"ID"},
   },
   want:    &TestRequest{ID: 123, Name: "Test"},
   wantErr: assert.NoError,
  },
  {
   name: "Missing body",
   args: args{
    w: httptest.NewRecorder(),
    r: func() *http.Request {
     req := httptest.NewRequest("POST", "/test/123", nil)
     req = testSetURLParam(req, "ID", "123")
     req.Header.Set("Content-Type", "application/json")
     return req
    }(),
    pathParams: []string{"ID"},
   },
   want:    nil,
   wantErr: assert.Error,
  },
  {
   name: "Missing Path Parameter",
   args: args{
    w: httptest.NewRecorder(),
    r: func() *http.Request {
     req := httptest.NewRequest("POST", "/test", nil)
     req.Header.Set("Content-Type", "application/json")
     return req
    }(),
    pathParams: []string{"ID"},
   },
   want:    nil,
   wantErr: assert.Error,
  },
  {
   name: "Invalid JSON Body",
   args: args{
    w: httptest.NewRecorder(),
    r: func() *http.Request {
     req := httptest.NewRequest("POST", "/test/123", bytes.NewBufferString("{invalid-json}"))
     req = testSetURLParam(req, "ID", "123")
     req.Header.Set("Content-Type", "application/json")
     return req
    }(),
    pathParams: []string{"ID"},
   },
   want:    nil,
   wantErr: assert.Error,
  },
  {
   name: "Validation Error for body",
   args: args{
    w: httptest.NewRecorder(),
    r: func() *http.Request {
     body, _ := json.Marshal(TestRequest{})
     req := httptest.NewRequest("POST", "/test/123", bytes.NewBuffer(body))
     req = testSetURLParam(req, "ID", "123")
     req.Header.Set("Content-Type", "application/json")
     return req
    }(),
    pathParams: []string{"ID"},
   },
   want:    nil,
   wantErr: assert.Error,
  },
  {
   name: "Validation Error for zero ID",
   args: args{
    w: httptest.NewRecorder(),
    r: func() *http.Request {
     body, _ := json.Marshal(TestRequest{Name: "Test"})
     req := httptest.NewRequest("POST", "/test/0", bytes.NewBuffer(body))
     req = testSetURLParam(req, "ID", "0")
     req.Header.Set("Content-Type", "application/json")
     return req
    }(),
    pathParams: []string{"ID"},
   },
   want:    nil,
   wantErr: assert.Error,
  },
 }

 for _, tt := range tests {
  t.Run(tt.name, func(t *testing.T) {
   got, err := ParseRequest[*TestRequest](tt.args.w, tt.args.r, tt.args.pathParams...)
   if !tt.wantErr(t, err, fmt.Sprintf("parseRequest(%v, %v, %v)", tt.args.w, tt.args.r, tt.args.pathParams)) {
    return
   }
   assert.Equalf(t, tt.want, got, "parseRequest(%v, %v, %v)", tt.args.w, tt.args.r, tt.args.pathParams)
  })
 }
}

As you can see, the Request layer uses the Validation layer. However, I still want to keep the layers separated in the code, not only to make it easier to maintain, but 'cause I may also want to use the validation layer isolated.

Depending on the needs, in the future, I may decide to keep all the layers isolated and allowing its co-dependency by using some interfaces.

Step 2: Implementing the Validation Layer

Once the request parsing is separated, we create a standalone validation function or module that handles the validation logic. By isolating this logic, we can easily test it and apply consistent validation rules across multiple endpoints.

For that, let's create the httpsuite/validation.go file:

package httpsuite

import (
 "errors"
 "github.com/go-playground/validator/v10"
)

// ValidationErrors represents a collection of validation errors for an HTTP request.
type ValidationErrors struct {
 Errors map[string][]string `json:"errors,omitempty"`
}

// NewValidationErrors creates a new ValidationErrors instance from a given error.
// It extracts field-specific validation errors and maps them for structured output.
func NewValidationErrors(err error) *ValidationErrors {
 var validationErrors validator.ValidationErrors
 errors.As(err, &validationErrors)

 fieldErrors := make(map[string][]string)
 for _, vErr := range validationErrors {
  fieldName := vErr.Field()
  fieldError := fieldName + " " + vErr.Tag()

  fieldErrors[fieldName] = append(fieldErrors[fieldName], fieldError)
 }

 return &ValidationErrors{Errors: fieldErrors}
}

// IsRequestValid validates the provided request struct using the go-playground/validator package.
// It returns a ValidationErrors instance if validation fails, or nil if the request is valid.
func IsRequestValid(request any) *ValidationErrors {
 validate := validator.New(validator.WithRequiredStructEnabled())
 err := validate.Struct(request)
 if err != nil {
  return NewValidationErrors(err)
 }
 return nil
}

Now, create the test file httpsuite/validation_test.go:

package httpsuite

import (
 "github.com/go-playground/validator/v10"
 "testing"

 "github.com/stretchr/testify/assert"
)

type TestValidationRequest struct {
 Name string `validate:"required"`
 Age  int    `validate:"required,min=18"`
}

func TestNewValidationErrors(t *testing.T) {
 validate := validator.New()
 request := TestValidationRequest{} // Missing required fields to trigger validation errors

 err := validate.Struct(request)
 if err == nil {
  t.Fatal("Expected validation errors, but got none")
 }

 validationErrors := NewValidationErrors(err)

 expectedErrors := map[string][]string{
  "Name": {"Name required"},
  "Age":  {"Age required"},
 }

 assert.Equal(t, expectedErrors, validationErrors.Errors)
}

func TestIsRequestValid(t *testing.T) {
 tests := []struct {
  name           string
  request        TestValidationRequest
  expectedErrors *ValidationErrors
 }{
  {
   name:           "Valid request",
   request:        TestValidationRequest{Name: "Alice", Age: 25},
   expectedErrors: nil, // No errors expected for valid input
  },
  {
   name:    "Missing Name and Age below minimum",
   request: TestValidationRequest{Age: 17},
   expectedErrors: &ValidationErrors{
    Errors: map[string][]string{
     "Name": {"Name required"},
     "Age":  {"Age min"},
    },
   },
  },
  {
   name:    "Missing Age",
   request: TestValidationRequest{Name: "Alice"},
   expectedErrors: &ValidationErrors{
    Errors: map[string][]string{
     "Age": {"Age required"},
    },
   },
  },
 }

 for _, tt := range tests {
  t.Run(tt.name, func(t *testing.T) {
   errs := IsRequestValid(tt.request)
   if tt.expectedErrors == nil {
    assert.Nil(t, errs)
   } else {
    assert.NotNil(t, errs)
    assert.Equal(t, tt.expectedErrors.Errors, errs.Errors)
   }
  })
 }
}

Step 3: Building the Response Layer

Finally, we refactor the response construction into a separate module. This ensures that all responses follow a consistent format, making it simpler to manage and debug responses throughout the application.

Create the file httpsuite/response.go:

package httpsuite

import (
 "encoding/json"
 "log"
 "net/http"
)

// Response represents the structure of an HTTP response, including a status code, message, and optional body.
type Response[T any] struct {
 Code    int    `json:"code"`
 Message string `json:"message"`
 Body    T      `json:"body,omitempty"`
}

// Marshal serializes the Response struct into a JSON byte slice.
// It logs an error if marshalling fails.
func (r *Response[T]) Marshal() []byte {
 jsonResponse, err := json.Marshal(r)
 if err != nil {
  log.Printf("failed to marshal response: %v", err)
 }

 return jsonResponse
}

// SendResponse creates a Response struct, serializes it to JSON, and writes it to the provided http.ResponseWriter.
// If the body parameter is non-nil, it will be included in the response body.
func SendResponse[T any](w http.ResponseWriter, message string, code int, body *T) {
 response := &Response[T]{
  Code:    code,
  Message: message,
 }
 if body != nil {
  response.Body = *body
 }

 writeResponse[T](w, response)
}

// writeResponse serializes a Response and writes it to the http.ResponseWriter with appropriate headers.
// If an error occurs during the write, it logs the error and sends a 500 Internal Server Error response.
func writeResponse[T any](w http.ResponseWriter, r *Response[T]) {
 jsonResponse := r.Marshal()

 w.Header().Set("Content-Type", "application/json")
 w.WriteHeader(r.Code)

 if _, err := w.Write(jsonResponse); err != nil {
  log.Printf("Error writing response: %v", err)
  http.Error(w, "Internal Server Error", http.StatusInternalServerError)
 }
}

Create the test file httpsuite/response_test.go:

package httpsuite

import (
 "net/http"
 "net/http/httptest"
 "testing"

 "github.com/stretchr/testify/assert"
)

type TestResponse struct {
 Key string `json:"key"`
}

func TestResponse_Marshal(t *testing.T) {
 tests := []struct {
  name     string
  response Response[any]
  expected string
 }{
  {
   name:     "Basic Response",
   response: Response[any]{Code: 200, Message: "OK"},
   expected: `{"code":200,"message":"OK"}`,
  },
  {
   name:     "Response with Body",
   response: Response[any]{Code: 201, Message: "Created", Body: map[string]string{"id": "123"}},
   expected: `{"code":201,"message":"Created","body":{"id":"123"}}`,
  },
  {
   name:     "Response with Empty Body",
   response: Response[any]{Code: 204, Message: "No Content", Body: nil},
   expected: `{"code":204,"message":"No Content"}`,
  },
 }

 for _, tt := range tests {
  t.Run(tt.name, func(t *testing.T) {
   jsonResponse := tt.response.Marshal()
   assert.JSONEq(t, tt.expected, string(jsonResponse))
  })
 }
}

func Test_SendResponse(t *testing.T) {
 tests := []struct {
  name           string
  message        string
  code           int
  body           any
  expectedCode   int
  expectedBody   string
  expectedHeader string
 }{
  {
   name:           "200 OK with TestResponse body",
   message:        "Success",
   code:           http.StatusOK,
   body:           &TestResponse{Key: "value"},
   expectedCode:   http.StatusOK,
   expectedBody:   `{"code":200,"message":"Success","body":{"key":"value"}}`,
   expectedHeader: "application/json",
  },
  {
   name:           "404 Not Found without body",
   message:        "Not Found",
   code:           http.StatusNotFound,
   body:           nil,
   expectedCode:   http.StatusNotFound,
   expectedBody:   `{"code":404,"message":"Not Found"}`,
   expectedHeader: "application/json",
  },
 }

 for _, tt := range tests {
  t.Run(tt.name, func(t *testing.T) {
   recorder := httptest.NewRecorder()

   switch body := tt.body.(type) {
   case *TestResponse:
    SendResponse[TestResponse](recorder, tt.message, tt.code, body)
   default:
    SendResponse(recorder, tt.message, tt.code, &tt.body)
   }

   assert.Equal(t, tt.expectedCode, recorder.Code)
   assert.Equal(t, tt.expectedHeader, recorder.Header().Get("Content-Type"))
   assert.JSONEq(t, tt.expectedBody, recorder.Body.String())
  })
 }
}

func TestWriteResponse(t *testing.T) {
 tests := []struct {
  name         string
  response     Response[any]
  expectedCode int
  expectedBody string
 }{
  {
   name:         "200 OK with Body",
   response:     Response[any]{Code: 200, Message: "OK", Body: map[string]string{"id": "123"}},
   expectedCode: 200,
   expectedBody: `{"code":200,"message":"OK","body":{"id":"123"}}`,
  },
  {
   name:         "500 Internal Server Error without Body",
   response:     Response[any]{Code: 500, Message: "Internal Server Error"},
   expectedCode: 500,
   expectedBody: `{"code":500,"message":"Internal Server Error"}`,
  },
 }

 for _, tt := range tests {
  t.Run(tt.name, func(t *testing.T) {
   recorder := httptest.NewRecorder()

   writeResponse(recorder, &tt.response)

   assert.Equal(t, tt.expectedCode, recorder.Code)
   assert.Equal(t, "application/json", recorder.Header().Get("Content-Type"))
   assert.JSONEq(t, tt.expectedBody, recorder.Body.String())
  })
 }
}

Each step of this refactoring allows us to simplify the handler logic by delegating specific responsibilities to well-defined layers. While I won’t show the complete code at every step, these changes involve moving parsing, validation, and response logic into their respective functions or files.

Refactoring the example code

Now, what we need is to change the old code to use the layers and let’s see how it will look like.

package main

import (
 "github.com/go-chi/chi/v5"
 "github.com/go-chi/chi/v5/middleware"
 "github.com/rluders/httpsuite"
 "log"
 "net/http"
)

type SampleRequest struct {
 Name string `json:"name" validate:"required,min=3"`
 Age  int    `json:"age" validate:"required,min=1"`
}

func (r *SampleRequest) SetParam(fieldName, value string) error {
 switch fieldName {
 case "name":
  r.Name = value
 }
 return nil
}

func main() {
 r := chi.NewRouter()
 r.Use(middleware.Logger)
 r.Use(middleware.Recoverer)

 r.Post("/submit/{name}", func(w http.ResponseWriter, r *http.Request) {
  // Step 1: Parse the request and validate it
  req, err := httpsuite.ParseRequest[*SampleRequest](w, r, "name")
  if err != nil {
   log.Printf("Error parsing or validating request: %v", err)
   return
  }

  // Step 2: Send a success response
  httpsuite.SendResponse(w, "Request received successfully", http.StatusOK, &req)
 })

 log.Println("Starting server on :8080")
 http.ListenAndServe(":8080", r)
}

By refactoring the handler code into layers for request parsing, validation, and response formatting, we have successfully removed the repetitive logic that was previously embedded within the handler itself. This modular approach not only improves readability but also enhances maintainability and testability by keeping each responsibility focused and reusable. With the handler now simplified, developers can easily understand and modify specific layers without affecting the entire flow, creating a cleaner, more scalable codebase.

Conclusion

I hope this step-by-step guide on structuring your Go microservices with dedicated request, validation, and response layers has provided insight into creating cleaner and more maintainable code. I’d love to hear your thoughts about this approach. Am I missing something important? How would you extend or improve this idea in your own projects?

I encourage you to explore the source code and use httpsuite directly in your projects. You can find the library in the rluders/httpsuite repository. Your feedback and contributions would be invaluable to make this library even more robust and useful for the Go community.

See you all in the next one.