Handling Eventual Consistency in Webhook

What is eventual consistency?

Eventual consistency is a consistency model used in distributed computing to achieve high availability and fault tolerance. It ensures that, given enough time and the absence of new updates, all copies of a data item will converge to the same value.

This usually happens when you have more than one node for your database, when one acts as a primary and the other is the secondary. Like the below image.

You write to the primary instance and read from the secondary instance to scale reads. The problem is that you can read data that is as fresh as in the primary until the data is fully replicated.
This is a common tradeoff when working in a distributed system.

Eventual consistent at Webhook

Some applications can send a webhook before the replication happens to all nodes of the database, and if the webhook payload does not have all the data that you need, you need to perform a GET in an endpoint to get the missing data. The pattern as below:

When doing a GET you can receive a 400 or 404 that the resource you are trying to GET does not exist, even if you already received the webhook with the given resource. This happens because of eventual consistency.

One solution to this is to retry this GET in an async job after some time, until the replication is done. The code below is a simple implementation in node using bulljs.

import Queue from 'bull';

const queue = new Queue('WEBHOOK');

const result = await resourceGet(webhook.resource.id);

if (result.error) {
  await queue.add(
    'PROCESS_WEBHOOK',
    { webhook },
    {
      attempts: 7,
      backoff: {
        type: 'exponential',
        delay: 10000,
      },
    }
  );

  return;
}

In Conclusion

Eventual consistency is a common topic of distributed systems that will affect your internal system and also your interaction with external systems.
Understanding why an error happens in production using observability tools is important to provide the best solution for your specific use case.

---

Woovi is an innovative startup revolutionizing the payment landscape. With Woovi, shoppers can enjoy the freedom to pay however they prefer. Our cutting-edge platform provides instant payment solutions, empowering merchants to accept orders and enhance their customer experience seamlessly.

If you're interested in joining our team, we're hiring! Check out our job openings at Woovi Careers.

---

Image by freepik