Circuit Breaker Pattern -Building Resilience

Hi devs,

When building microservices architectures, one of the key challenges is managing failures gracefully. Services often depend on other services to function, and any dependency can potentially fail. Circuit Breaker is a design pattern that helps improve resilience by detecting failures and preventing continuous attempts to reach an unresponsive service, which can lead to cascading failures and degraded performance across the system.

Let’s dive into how the Circuit Breaker pattern works, when to use it, and how to implement it effectively.

What Is the Circuit Breaker Pattern?

The Circuit Breaker pattern is inspired by electrical circuit breakers: if there’s a fault in one part of a system, the circuit “breaks,” or opens, to prevent further damage. In microservices, the pattern is implemented by “wrapping” a function call to an external service with a logic that monitors for failures and decides when to stop sending requests temporarily.

In practice, the Circuit Breaker pattern has three states:

1. Closed: The service call works as expected. Requests are sent through normally.
2. Open: The service is considered unhealthy, and requests to it are immediately blocked to avoid overwhelming it or the system.
3. Half-Open: After a defined period, a limited number of requests are allowed to test if the service has recovered.

Why Use the Circuit Breaker Pattern?

Using a Circuit Breaker pattern can prevent cascading failures and help maintain a healthy, responsive system. Here’s a breakdown of its key benefits:

• Improved System Stability: By blocking requests to a failing service, the pattern prevents wasted resources and potential service degradation.
• Reduced Latency: The pattern prevents waiting for timeouts when services are unresponsive, allowing your application to respond faster.
• Failover Strategies: Circuit Breakers can allow you to implement fallback responses or reroute requests to alternative services.
• Better User Experience: By avoiding timeouts and endless retries, Circuit Breakers can help maintain a consistent and responsive user experience, even during partial outages.

How It Works in Practice

Imagine an e-commerce application with separate services for User, Orders, and Inventory. The Orders service needs to access the Inventory service to check product availability before confirming an order. If the Inventory service is down or slow, it can delay or break the order confirmation process.

Using the Circuit Breaker pattern, the Orders service can “break” its connection to the Inventory service if too many requests fail or time out, and try again after a specified time. In the meantime, you could configure it to send a fallback response, notifying users of a temporary issue or redirecting them to a cache.

Implementing Circuit Breaker in C

Let’s look at a simple example of how to implement the Circuit Breaker pattern in C# using the Polly library, a popular .NET library for handling resilience and transient-fault-handling.

Step 1: Install Polly

First, install the Polly library using NuGet:

dotnet add package Polly

Step 2: Set Up the Circuit Breaker Policy

Define a circuit breaker policy with Polly:

using Polly;
using Polly.CircuitBreaker;

// Define a Circuit Breaker policy
var circuitBreakerPolicy = Policy
    .Handle<Exception>() // Handle all exceptions (or specific ones)
    .CircuitBreaker(
        exceptionsAllowedBeforeBreaking: 3, // Number of exceptions before opening the circuit
        durationOfBreak: TimeSpan.FromSeconds(30), // Duration of the break
        onBreak: (ex, breakDelay) => {
            Console.WriteLine($"Circuit broken! Break for {breakDelay.TotalSeconds} seconds.");
        },
        onReset: () => {
            Console.WriteLine("Circuit closed again, requests are allowed.");
        },
        onHalfOpen: () => {
            Console.WriteLine("Circuit half-open, testing for recovery.");
        });

Step 3: Use the Circuit Breaker Policy with a Service Call

Wrap your external service calls (e.g., to the Inventory service) in the circuit breaker policy:

public async Task CheckInventoryAsync()
{
    try
    {
        // Execute a function that calls the Inventory service within the circuit breaker
        await circuitBreakerPolicy.ExecuteAsync(async () =>
        {
            // Simulate service call here
            Console.WriteLine("Calling Inventory Service...");
            // Simulate a failure
            throw new Exception("Inventory service unavailable!");
        });
    }
    catch (BrokenCircuitException)
    {
        // Handle the case when the circuit is open and requests are blocked
        Console.WriteLine("Circuit is open, fallback action here.");
    }
}

Circuit Breaker States in Action

Here's how the circuit breaker will work in this example:

• Closed: The first three failed attempts will trigger retries.
• Open: After three failures, the circuit will open, preventing further calls for 30 seconds.
• Half-Open: After 30 seconds, the circuit will enter a half-open state and test the service with limited requests. If successful, it will reset to a closed state; if it fails, it will reopen.

When to Use Circuit Breaker (and When Not To)

The Circuit Breaker pattern is ideal for microservices architectures where you need to protect services from cascading failures, especially in cases of:

• External API calls: Protect your service from failures in third-party APIs.
• Database connections: Prevent repeated attempts to reach an unreachable database.
• Inter-service calls: Manage dependencies between internal services gracefully.

However, Circuit Breaker is not a one-size-fits-all solution. If your application requires constant connectivity (such as real-time streaming), or if you can afford retries rather than failures, then the Circuit Breaker might not be ideal.

The Circuit Breaker pattern is an essential tool for building resilient microservices systems, providing a way to handle transient failures and prevent cascading service breakdowns. With tools like Polly in C#, implementing circuit breakers has become straightforward, making it a valuable addition to any developer's toolkit.