Event-driven design with Log Events and RabbitMQ in Golang

The adoption of event-driven architecture is on the rise as teams pursue more adaptable, scalable, and agile solutions to meet the requirements of contemporary applications. Event-driven architectures support real-time updates and streamline integration across different systems by enabling communication through standardized and structured events.

In a previous blog post, I discussed how webhooks in Auth0 can transmit events, thereby leveraging these events to initiate logic execution. In this article, I will delve into the technical aspects of this architecture and demonstrate how Go (Golang) can be utilized to construct such a system.

Main components:

Let's first have a look at the main components that drive this system.

Log Events:

Auth0 has log events associated with every activity at a tenant level. These events can be used for monitoring or audit purposes. The codes for each event can be checked out here

Webhooks:

We use auth0 webhooks to deliver filtered events to our producer. We filter these events since we are interested in only a handful.

RabbitMQ

RabbitMQ supports multiple messaging protocols, and the one we use to route messages is the Advanced Messaging Queueing Protocol (AMQP). AMQP has three main entities – Queues, Exchanges and Bindings.

Behind the scenes

When an event is triggered in Auth0, it's immediately sent via webhook to our publisher, which then publishes it based on the event type. Once published, the event goes to an exchange. The exchange directs the message to connected queues, where consumers receive it. To enable this process, we establish a channel. This channel allows us to publish messages to exchange and declare queues for subscription.

To create a new queue, we utilize the QueueDeclare function provided by the package on the channel, specifying our desired queue properties. With the queue created, we can use the channel's Publish function to send a message.

Next, we create a consumer that connects to our RabbitMQ and establishes a channel for communication. Using this channel, we can consume messages using the Consume method defined for it.

Groundwork

We use golang-auth0 management package to work on the log events and for the queue actions we use github.com/rabbitmq/amqp091-go.

Given below are the snippets:

Publishing:

A detailed structure of the log can be found here

    for _, auth0log := range payload.Logs {

        switch auth0log.Data.Type {
        case "slo":
            _, err = c.Publish(ctx, PublishRequest{
                ---your logic---
            })

        case "ss":
            _, err = c.Publish(ctx,PublishRequest{
                    -- your logic -----
                })

        }
    }

Exchange:

    if consumeOptions.BindingExchange != "" {
        for _, routingKey := range routingKeys {
            err = consumer.chManager.channel.QueueBind(
                queue,
                routingKey,
                consumeOptions.BindingExchange,
                consumeOptions.BindingNoWait,
                tableToAMQPTable(consumeOptions.BindingArgs),
            )
            if err != nil {
                return err
            }
        }
    }

Consuming:

func (c *Client) Consume() {
    err := c.consumer.StartConsuming(
        func(ctx context.Context, d queue.Delivery) bool {
            err := c.processMessages(ctx, d.Body, d.Exchange)
            if err != nil {
                c.log.Error().Ctx(ctx).Err(err).Str("exchange", d.Exchange).Msg("failed publishing")
                return nack // send to dlx
            }
            return ack // message acknowledged
        },
        c.queueName,
        []string{c.queueRoutingKey},
        func(opts *queue.ConsumeOptions) {
            opts.BindingExchange = c.queueBidingExchange
            opts.QueueDurable = true
            opts.QueueArgs = map[string]interface{}{
                "x-dead-letter-exchange": c.queueBidingExchangeDlx,
            }
        },
    )
    if err != nil {
        c.log.Fatal().Err(err).Msg("consumer: Failed to StartConsuming")
    }

    // block main thread so consumers run forever
    forever := make(chan struct{})
    <-forever
}   

Thus, by leveraging webhooks in Auth0 to trigger events and employing RabbitMQ for reliable message queuing and delivery, we can build scalable and responsive applications. This approach not only enhances flexibility but also supports seamless event processing, enabling efficient handling of asynchronous operations.

I hope this article was helpful and can prove to be beneficial in your event-driven journey.

Happy coding :)