Azure Event Grid series: handling big events

This is a series of blogs to talk and discuss about good practices and tips for Azure Event Grid. Some of the topics that will be discussed can be applied not only to Event Grid but also to any other event/message based service.

TL;DR

Claim Check Pattern is a widely used pattern to keep events and messages small in order to make them fit into the service size limits. The idea is to use an intermediate storage to save the event/message payload and send the event/message with the stored reference.

Lightweight events

According to the official Microsoft documentation, an event is a lightweight notification of a condition or a state change. Accordingly, Event Grid limits the event's size to 1 MB and are billed on 64 KB slices. Therefore, if we want to keep things cost efficient the idea is to keep events light and under the limit.

But, what does it happen when you need to send some piece of information larger than the service limit?

Claim Check Pattern

The idea behind this pattern is to keep the event light storing the event payload into an external data store and send the reference to the stored payload on the event. Then, the subscriber (the service that receives the event) can download the payload if it is necessary.

For a given event, the pattern can be applied following two different approaches:

Original Event:

{
    "id": "c4a9175a-2f0e-11eb-adc1-0242ac120002",
    "eventType": "NewBlogPost",
    "time": "2020-10-15T14:11:39.2329267Z",
    "data": {
        "title": "My new blog",
        "text": "Really big text ..."
    }
}

1. Store the whole event including metadata and payload. Event metadata is formed by all the common fields that are shared between different events in an eco-system, for example the eventType or the eventTime. The sent event will contain just a reference to the stored information.

{
    "reference": "https://my-external-data-store.com/mydata/c4a9175a-2f0e-11eb-adc1-0242ac120002"
}

1. Store only the event payload and keep the event with its metadata. The sent event will contain the metadata and a reference to the stored payload.

{
    "id": "c4a9175a-2f0e-11eb-adc1-0242ac120002",
    "eventType": "NewBlogPost",
    "time": "2020-10-15T14:11:39.2329267Z",
    "reference": "https://my-external-data-store.com/mydata/c4a9175a-2f0e-11eb-adc1-0242ac120002"
}

On one hand, first approach creates a lighter event, however it forces every subscriber to download the event from the external data store. On the other hand, second approach lets subscribers to decide whether they need to download the event or not, depending on the event metadata.

Imagine a case where you have two subscribers for the same event (a new blog entry has been created). First subscriber is in charge of having a blog entries counter, so it won't need to download the event payload from the external data store. On the other hand, second subscriber must create a summary for each blog entry, in that case it will need to download the event payload to fulfill its use case.

Removing the event

When using this pattern is important to remove the event from the storage once it has been consumed. If sender-subscriber is always a one to one relationship, we can keep things simple moving the removal logic into the subscriber, once it finishes processing the event it can remove safely the event from the storage.

However, when using a event driven architecture it is more than usual to have a one to many relationship, so the subscribers can't remove the event or will make impossible to process the event for other subscribers. The solution is to remove asynchronously the event data by an independent service. For example, if you are using Blob Storage to store the event data, this removal can be done easily with a delete policy rule.

Conditional Claim Check Pattern

It is a small improvement to the original idea. In this case, the pattern will be applied only to those heavy events that do not fit the limits. So, instead of storing every single event into the external data store, it is necessary to check first the event size and use the storage only for those large events that do not fit into the limits.

Main advantage of this approach is that it keeps things cost efficient. Cloud data stores are usually expensive so the cost can be cut down if we reduce the number of events that use the storage. However, some extra logic is necessary on both sender and subscriber. On one side, sender must check the event size before sending it. On the other side, subscriber must verify if the received event is a complete one or must be download.

Code samples

You can find in this repository a code sample using Azure Functions, Event Grid and Blob Storage to apply the Claim Check Pattern: https://github.com/DavidGSola/event-grid-good-practices/tree/main/claim-check