How to Build a Fault-Tolerant Microservice for Payment Retries

Imagine you have a loyal customer, Abubakar, who subscribes to your monthly service and has always had a smooth experience. But payment failures like network issues, gateway timeout, or system hiccups can interrupt transactions at any time. Without a retry mechanism, these failures can block access, frustrate users, and result in lost revenue.

Automated payments are the revenue lifeline for subscription-based businesses, so building a dedicated retry service for failed transactions is important. This service gives failed payments another chance (or two or three) before they're truly lost. In this guide, you’ll learn how to design a fault-tolerant microservice that uses message queues, exponential backoff, and concurrency handling to recover from payment failures gracefully.

Before diving into the implementation details, let’s explore the importance of payment retries.

Why do you Need Payment Retries?

Payment gateways can fail for many reasons, such as temporary network issues, bank downtime, or rate limits. Without a retry mechanism, these failures can result in involuntary customer churn. Below are some of the reasons why retries are essential for your business:

• Improves Customer Experience: Automated payment allows your customers to get around re-entering payment details or get in contact with the support team for assistance. This process saves time, reduces frustration, and improves the overall experience.
• Make Informed Decisions: With payment retries, you gain access to failed payment data, allowing you to analyze patterns and identify the best times for successful transactions. This enables you to implement smart retry strategies, improving the success rate.
• Prevents Service Disruptions: For subscription-based businesses, uninterrupted service is crucial. Payment retries help you avoid disruptions caused by failed transactions, ensuring customers continue to receive the service they expect.
• Reduces Operation Costs: Payment retries save you from costly administrative interventions like chargeback processing, support services, etc.
• Increases Transaction Success Rates: Retries give your business another chance to process failed transactions, improving overall payment success rates and enhancing the efficiency of the payment system.
• Maintains Consistent Cash Flow: Retries help your business maintain consistent cash flow by recovering revenues from transactions that initially failed.

Designing the Retry Service

To implement retry logic, you need a message queue. A message queue is a tool that lets independent applications and services exchange information. In our case, we'll use Redis as the message queue to store references to failed transactions until they are processed and removed. Redis is an excellent database for managing multiple tasks that are lined up for processing.

This implementation uses Go and Docker (for running Redis). You can also use a managed Redis server to process your queue. While Go is used here, the approach applies to any other programming language of your choice.

To get started, install Redis in Docker using the command below:

docker run --name redis -d -p 6379:6379 redis

With Redis up and running, push failed transactions into the Redis queues for retries.

package main

import (
    "context"
    "fmt"
    "github.com/redis/go-redis/v9"
    "log"
)

var redisClient = redis.NewClient(&redis.Options{
    Addr: "localhost:6379",
})

func queueFailedPayment(txID string) error {
    ctx := context.Background()
    err := redisClient.LPush(ctx, "failed_payments", txID).Err()
    if err != nil {
        log.Printf("Failed to queue transaction %s: %v", txID, err)
    }
    return err
}

This code uses a Redis List to save failed transactions, which works well for simple queueing and smaller volumes of failures. However, if you expect failures to occur on a larger scale or need more advanced features, using a Redis Stream would be a better fit.

Implementing Exponential Backoff

Now that the failed transactions are saved in Redis, you might think it's best to retry them immediately. However, that's not the case. Instead, you should use exponential backoff—a strategy that gradually increases the wait time between retries to avoid overloading the server.

Exponential backoff uses a formula to calculate the retries as follows: delay = base_delay * 2^(retry_attempt)

For example, if the base delay is 2 seconds, the retry delay for each attempt would be calculated as follows:

Retry Attempt	Calculation	Delay (seconds)
1	2×2^1=4	4
2	2×2^2=8	8
3	2×2^3=16	16
4	2×2^4=32	32

import (
    "math"
    "time"
)

func exponentialBackoff(attempt int) time.Duration {
    base := 2 // Base time in seconds
    maxDelay := 60 // Max cap in seconds

    delay := time.Duration(math.Min(float64(base*int(math.Pow(2, float64(attempt)))), float64(maxDelay))) * time.Second
    return delay
}

Process Failed Payments Concurrently

Next, you need to create a worker that fetches failed transactions and uses the exponential backoff to retry the payment without overloading the system.

func processFailedPayments() {
    ctx := context.Background()
    for {
        txID, err := redisClient.RPop(ctx, "failed_payments").Result()
        if err != nil {
            time.Sleep(5 * time.Second) // No failed payments, wait before retrying
            continue
        }

        for attempt := 0; attempt < 5; attempt++ { // Retry up to 5 times
            success := attemptPayment(txID) // A call to your payment gateway
            if success {
                fmt.Println("Payment successful for", txID)
                break
            }
            time.Sleep(exponentialBackoff(attempt))
        }
    }
}

Lastly, limit the number of retries by processing the payment concurrently.

// ensure concurrent transactions execute appropriately
func startWorkers(workerCount int) {
    for i := 0; i < workerCount; i++ {
        go processFailedPayments()
    }
}

Handling Idempotency

Idempotency means that making the same request multiple times won’t trigger duplicate operations. When handling payments, you need to check if a payment has already been processed before retrying. This helps prevent customers from being charged more than once.

func attemptPayment(txID string) bool {
    // Check if transaction was already processed to avoid saving the same data multiple time.
    if isPaymentSuccessful(txID) {
        return true
    }

    // Retry payment to Payment Gateway.
    response := makePaymentRequest(txID)
    if response.Status == "success" {
        markPaymentAsProcessed(txID)
        return true
    }
    return false
}

These steps help you build a more resilient payment system that allows transactions to go through even when there are network failures or API issues. This improves reliability and supports continuous revenue collection.

While a retry service offers many benefits and helps reduce customer churn, it also comes with technical challenges, such as:

• High resource consumption.
• The need for advanced monitoring of failed transactions.
• Optimizing retry limits based on the payment gateway.
• Requires extra technical effort to build a circuit breaker pattern (a system that stops retrying after several failed attempts).

You can address these challenges by using a payment gateway like Flutterwave, which reduces the technical overhead and helps you build better retry service.

How Flutterwave can Help you Build a Better Retry Service

Flutterwave provides built-in tools to help your businesses recover lost revenue, manage failed payments, and automate retries. Here’s how it can help:

Checkout Timeout and Retries

Flutterwave Checkout offers more than customization for amounts, branding, and payment options. It also lets you control how long customers have to complete a payment and how many times they can retry.

For example, if you set a 5-minute timeout and allow five retry attempts, customers can attempt to complete their payment within this window before the transaction is considered failed. This built-in flexibility serves as the first fail-safe mechanism, giving customers a chance to finalize their payment before your retry service takes over.

<input type="hidden" name="public_key" value="SAMPLE PUBLIC KEY" />
<input type="hidden" name="tx_ref" value="bitethtx-019203" />
<input type="hidden" name="amount" value="3400" />
<input type="hidden" name="currency" value="NGN" />
<input
    type="hidden"
    name="redirect_url"
    value="https://demoredirect.localhost.me/"
/>
<input type="hidden" name="meta[token]" value="54" />
<input type="hidden" name="customer[name]" value="Jesse Pinkman" />
<input
    type="hidden"
    name="customer[email]"
    value="[e-mail]"
/>
<input type="hidden" name="configurations[session_duration]" value="10" />
#Session timeout in minutes (maxValue: 1440 minutes)
<input type="hidden" name="configurations[max_retry_attempt]" value="5" /> #Max
retry (int)

Robust API Suite

Flutterwave provides dedicated APIs, such as transaction verification and payment retry endpoints, to help manage failed transactions.

• The transaction verification endpoint allows you to check the real-time status of a transaction.
• The payment retry endpoint enables you to re-initiate a failed payment when necessary.

By integrating these APIs into your retry service, you can automate payment recovery and improve transaction success rates.

Customized Billing

With Flutterwave, you can customize your customers' billing cycles using either the recurring payment endpoint or the dashboard. This flexibility allows you to tailor your payment collection process to different customer needs, reducing the chances of failed transactions. As a result, your retry service is used less frequently, and resource usage is optimized.

Real-Time Dashboard

Flutterwave offers comprehensive analytics and reporting tools that provide real-time insights into:

• Failed payments and their causes
• Success rates of transactions
• Retry attempts and their outcomes
• Overall payment performance trends

These insights help you monitor transactions, optimize your retry strategy, and improve revenue recovery.

Wrapping Up

A payment retry service helps your business recover failed transactions by automatically retrying them on a scheduled basis. It helps resolve network errors, bank downtimes, and gateway failures by reducing revenue loss and improving the payment experience.

Flutterwave makes this process even more effective with:

• Checkout retry settings that give customers multiple chances to complete payments.
• Transaction verification and retry APIs that automate recovery.
• Customizable billing cycles that help reduce failed transactions.
• Real-time analytics that provide insights to optimize retry strategies.

With Flutterwave’s tools, you can build a smarter, more efficient, and automated payment retry system that improves transaction success rates and minimizes disruptions.

Get started with Flutterwave today.