Instrumentation for Event Driven

Woovi invests a lot in observability as discussed in this article https://dev.to/woovi/observability-with-elasticsearch-kibana-and-apm-3dhb.
In this article, we explain how we instrument our events in our event-driven architecture.

Observability Concepts

Before dive in, let's define a few basic observability concepts.

A Transaction represents one full request, one job, one unit of work. All POST /charge will have the same transaction type and name. A Transaction has a start and an end.

A Span represents a unit of work inside a transaction. A Span has a start and an end. Examples of spans: database query, http request, redis request, etc.

A Label is a metadata attached to a transaction or span.

Event-Driven Instrumentation

We use Elastic APM as our Application Performance Monitoring.
elastic-apm-node auto instrument many packages like koa, mongodb and redis, but we also need to manually instrument to address Woovi needs.

We use bull-js for our distributed queue and event-driven library.

To register a function as a handler for an event, we do this in bull-js

const queue = new Queue('WEBHOOK', config.REDIS_HOST)

queue.process('CHARGE_PAID', chargePaidJob);

We would like to create one transaction per job execution.
APM Server and Kibana will group the same transaction type and name.

To make this possible, we create a higher order function that start the transaction, set some job data as labels/metadata, mark the transaction as a success or failure and end the transaction after the job is done.
The instrumentation function is below:

export const jobMiddleware = (fn: (job: Job) => Promise<any>) => {
  const type = 'job';

  const middleware = async (...args: any[]) => {
    const [job] = args;

    const name = job.name;
    const trans = apm.startTransaction(name, type);

    if (job.data) {
      apm.setLabel('jobData', JSON.stringify(job.data));
    }

    let result;

    try {
      result = await fn(...args);

      if (trans) {
        trans.setOutcome('success');
      }
    } catch (err) {
      apm.captureError(err);
      if (trans) {
        trans.setOutcome('failure');
      }
    }

    if (trans) {
      trans.end();
    }

    return result;
  }

  return middleware;
}

The result in Kibana is like this:

Each event/job has their own transaction, that we can deep dive to understand what is happening.
It is like seeing code being executed in production without needing to put any console.log on it.

In Conclusion

The right instrumentation enables observability in production.
This enables you to understand what your code is doing in production, without using console.log.
You can find slow queries and code, wrong approaches, many databases calls when just a few would be enough.
I hope this article helps you improve your instrumentation to ensure you have the right observability in place when you need to fix a bug in production.

---

Woovi
Woovi is a Startup that enables shoppers to pay as they like. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

---

Photo by Ellen Qin on Unsplash