Circuit Breaker 🎛️ pattern with GO

Problem

As Einstein said "perpetual motion machines are impossible".

Server ⚙️ can

• Go down i.e Go Offline
• Constantly give 500 i.e DBs they are hitting can go down
• N number of natural causes that can make a remote server useless.

Since Client won't have any information of the Server state, it will keep on trying and all the requests are gonna be failing.

Solution

Circuit Breaker proxy for Client

Microsoft's proposed solution is a Simple State Machine which can exist in one of three states

Closed 🟢

-- No issues, operations can go as is.

Open 🔴

-- Machine (i.e Server) is not in a good state, block everything.

Half Open 🟡

-- Testing fire, let one or more requestes to go through to check Machine state.
-- if success flip the machine state to "closed" 🟢
-- else move it to "open 🔴"

Implementation

• we can go through a well thought out implementation for this in Sony Circuit Breaker
• Below is an example for it.

Server

// server.go
package main

import (
    "log"
    "net/http"
    "os"
)

// ExampleServer is a test server to check the "CircuitBreaker" pattern
type ExampleServer struct {
    addr      string
    logger    *log.Logger
    isEnabled bool
}

// NewExampleServer creates the instance of our server
func NewExampleServer(addr string) *ExampleServer {
    return &ExampleServer{
        addr:      addr,
        logger:    log.New(os.Stdout, "Server\t", log.LstdFlags),
        isEnabled: true,
    }
}

// ListenAndServe starts listening on the address provided
// on creating the instance.
func (s *ExampleServer) ListenAndServe() error {
    // The main endpoint we will request to
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if s.isEnabled {
            s.logger.Println("responded with OK")
            w.WriteHeader(http.StatusOK)
        } else {
            s.logger.Println("responded with Error")
            w.WriteHeader(http.StatusInternalServerError)
        }
    })

    // Toggle endpoint to switch on and off responses from the main one
    http.HandleFunc("/toggle", func(w http.ResponseWriter, r *http.Request) {
        s.isEnabled = !s.isEnabled
        s.logger.Println("toggled. Is enabled:", s.isEnabled)
        w.WriteHeader(http.StatusOK)
    })

    return http.ListenAndServe(s.addr, nil)
}

Server.go

• struct "ExampleServer"

type ExampleServer struct {
    addr      string
    logger    *log.Logger
    isEnabled bool
}

• here "isEnabled" is the server's way of saying I am online / offline.
• when false don't send me anything at all, cause I am gonna fail them regardless.

ExampleServer servers two routes

• / -- responds with 200
• /toggle -- responds with 200 -- toggles "isEnabled" struct state.

client.go

// client.go
package main

import (
    "errors"
    "net/http"
)

type NotificationClient interface {
    Send() error // We ignore all the arguments to simplify the demo
}

type SmsClient struct {
    baseUrl string
}

func NewSmsClient(baseUrl string) *SmsClient {
    return &SmsClient{
        baseUrl: baseUrl,
    }
}

func (s *SmsClient) Send() error {
    url := s.baseUrl + "/"
    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return errors.New("bad response")
    }

    return nil
}

SmsClient struct has a method Send just makes an Api hit to server, return error when Api hit is not successfull.

func (s *SmsClient) Send() error {
    url := s.baseUrl + "/"
    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return errors.New("bad response")
    }

    return nil
}

circuit_breaker.go

// circuit_breaker.go
package main

import (
    "log"
    "os"
    "time"

    "github.com/sony/gobreaker/v2"
)

type ClientCircuitBreakerProxy struct {
    client NotificationClient
    logger *log.Logger
    gb     *gobreaker.CircuitBreaker[any] // downloaded lib structure
}

// shouldBeSwitchedToOpen checks if the circuit breaker should
// switch to the Open state
func shouldBeSwitchedToOpen(counts gobreaker.Counts) bool {
    failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
    return counts.Requests >= 3 && failureRatio >= 0.6
}

func NewClientCircuitBreakerProxy(client NotificationClient) *ClientCircuitBreakerProxy {
    logger := log.New(os.Stdout, "CB\t", log.LstdFlags)

    // We need circuit breaker configuration
    cfg := gobreaker.Settings{
        // When to flush counters int the Closed state
        Interval: 5 * time.Second,
        // Time to switch from Open to Half-open
        Timeout: 7 * time.Second,
        // Function with check when to switch from Closed to Open
        ReadyToTrip: shouldBeSwitchedToOpen,
        // set Max Request in Half Open state to 5
        MaxRequests: 5,
        // On State change Handler Fn
        OnStateChange: func(_ string, from gobreaker.State, to gobreaker.State) {
            // Handler for every state change. We'll use for debugging purpose
            logger.Println("state changed from", from.String(), "to", to.String())
        },
    }

    return &ClientCircuitBreakerProxy{
        client: client,
        logger: logger,
        gb:     gobreaker.NewCircuitBreaker[any](cfg),
    }
}

func (c *ClientCircuitBreakerProxy) Send() error {
    // We call the Execute method and wrap our client's call
    _, err := c.gb.Execute(func() (any, error) {
        err := c.client.Send()
        return nil, err
    })
    return err
}

struct ClientCircuitBreakerProxy

• holds client which can house SmsClient which we declared above.
• holds an instance for CircuitBreaker the state machine which hold the implementation for "Circuit Breaker" pattern.

type ClientCircuitBreakerProxy struct {
    client NotificationClient
    logger *log.Logger
    gb     *gobreaker.CircuitBreaker[any] // downloaded lib structure
}

In the implementation NewClientCircuitBreakerProxy fn takes a SmsClient as parameter and provides the inital values for CircuitBreaker

    // We need circuit breaker configuration
    cfg := gobreaker.Settings{
        // When to flush counters int the Closed state
        Interval: 5 * time.Second,
        // Time to switch from Open to Half-open
        Timeout: 7 * time.Second,
        // Function with check when to switch from Closed to Open
        ReadyToTrip: shouldBeSwitchedToOpen,
        // set Max Request in Half Open state to 5
        MaxRequests: 5,
        // On State change Handler Fn
        OnStateChange: func(_ string, from gobreaker.State, to gobreaker.State) {
            // Handler for every state change. We'll use for debugging purpose
            logger.Println("state changed from", from.String(), "to", to.String())
        },
    }

Send method in struct ClientCircuitBreakerProxy is a wrapper to client's Send method.

main.go

// main.go
package main

import (
    "log"
    "os"
    "time"
)

func main() {
    logger := log.New(os.Stdout, "Main\t", log.LstdFlags)
    server := NewExampleServer(":8080")

    go func() {
        _ = server.ListenAndServe()
    }()

    var client NotificationClient

    client = NewSmsClient("http://127.0.0.1:8080")

    client = NewClientCircuitBreakerProxy(client)

    for {
        err := client.Send()
        time.Sleep(1 * time.Second)
        if err != nil {
            logger.Println("caught an error", err)
        }
    }
}

• create a logger with

logger := log.New(os.Stdout, "Main\t", log.LstdFlags)

• start the simple go server with an run it concurrently with go routine

server := NewExampleServer(":8080")

    go func() {
        _ = server.ListenAndServe()
    }()

• Initiate the Client with

    var client NotificationClient

    client = NewSmsClient("http://127.0.0.1:8080")

• wrap it with Circuit Breaker proxy

    client = NewClientCircuitBreakerProxy(client)

• run an infinity loop which makes API hits to out simple server

    for {
        err := client.Send()
        time.Sleep(1 * time.Second)
        if err != nil {
            logger.Println("caught an error", err)
        }
    }

Conclusion

• As the Client makes the API hits in regular intervals to the Server
• till an API hit /toggle Circuit will make the Circuit Breaker to be in open 🟢

Reference

Circuit Breaker Pattern in GO

Code

Sandy10247 / go-breaker-example

A Simple implementation of Circuit Breaker pattern.

go-breaker-example

A Simple implementation of Circuit Breaker pattern.

execution

go run .

output

View on GitHub