My Journey in Open Source - async-task-packer

Link to the repo

What is it?

This is a zero-dependency and low-footprint library that allows you to pack a set of asynchronous tasks and execute them in a controlled way.

It allows you to execute those tasks in time-based intervals or in size-based chunks.

The library also allows you to indicate the order of execution of the packeted tasks (strict or loose).

It provides many other configurations you can leverage to better control the flow of the pack.

How do I install it?

You can install it by using the following command:

npm install async-task-packer

How can I use it?

Create a packer by calling the createPacker function and pass it the configuration you want to use.

Then, wrap your async functions with the resulting function to add them to the pack.

Use-case #1 - Delayed logger

const packer = createPacker({
  executionMethod: 'interval',
  executionType: 'loose',
  interval: 1000
});

const logger = (message) => {
  const createdAt = new Date();
  packer(() => {
    sendLogToAnExternalApi(message, createdAt);
  });
};

logger('Hello');
logger('World');

Use-case #2 - Analytics chunk-based packer

In this scenario, we want to send a batch of events to an external API.

We want to send the events in chunks of 10, but we also want to send the events after 10 seconds, even if the chunk is not full, to prevent losing important information.

We don't care about event ordering (loose type) as we provide a createdAt timestamp in the event payload and we can sort the events on the server side.

  const packer = createPacker({
    executionMethod: 'chunk',
    executionType: 'loose',
    chunkSize: 10,
    maxChunkLifetime: 10*1000 // Using a debounce mechanism, the packer will execute the pack after 10 seconds, even if it's not full
  });

  const track = (event, ...args) => {
    const createdAt = new Date();
    packer(() => {
      sendEventToAnExternalApi(event, createdAt, ...args);
    });
  };

  track('user_login', { userId: 123 });
  track('item_added_to_cart', { cartUUID: 'abc', itemId: 456 });
  track('item_added_to_cart', { cartUUID: 'abc', itemId: 789 });
  track('order_completed', { cartUUID: 'abc' });
  track('user_logout', { userId: 123 });

(More examples in the GitHub Repo)

API

The createPacker function accepts an object with the following properties:

• executionMethod (required): The execution method to use. It can be either chunk or interval.
  • If you choose chunk as the execution method, you can also provide the following properties:
  • chunkSize (required): The size of the chunk to use. It must be a positive integer.
  • maxChunkLifetime (optional): The maximum lifetime of a chunk. It must be a positive integer. Defaults to 600000 (10 minutes).
  • unref (optional): A boolean indicating whether the interval should be unrefed or not. Defaults to false. (https://nodejs.org/api/timers.html#timers_timeout_unref)
  • If you choose interval as the execution method, you can also provide the following properties:
  • interval (required): The interval to use. It must be a positive integer.
  • debounce (optional): A boolean indicating whether the interval should be debounced or not. Defaults to false. Setting it to true will cause the interval to be reset every time a new task is added to the pack.
  • unref (optional): A boolean indicating whether the interval should be unrefed or not. Defaults to false. (https://nodejs.org/api/timers.html#timers_timeout_unref)
• executionType (required): The execution type to use. It can be either strict or loose.
  • If you choose strict as the execution type, the packer will execute the tasks in the order they were added to the pack.
  • If you choose loose as the execution type, the packer will execute the tasks in parallel and will not enforce the order they were added to the pack.

To better control the pack flow, you can also pass the following properties:

• expectResolutions (optional): A boolean indicating whether the packer should expect the tasks to be resolved or not. Defaults to false.
  • If set to false, the packer will consider the pack as resolved as soon as the tasks are completed, regardless of whether they are resolved or rejected. Setting this option to false still allows you to catch errors by providing an onCatch method.
  • If set to true, the packer will consider the pack as resolved only if all the tasks are resolved. If one of the tasks is rejected, the packer will consider the pack as rejected and will call the packer onCatch method if provided or will throw the error if no onCatch method is provided.
• awaitAllTasks (optional): A boolean indicating whether the packer should wait for all the tasks to be executed before resolving the pack. Defaults to false.
  • If set to false, the packer will resolve the pack as soon as the tasks are executed.
  • If set to true, the packer will wait for all the tasks to be executed before resolving the pack. Please note that this option will immediately fill the pack with the tasks currently in the queue if conditions are met. It means a new pack could potentially be created and executed even if the previous one is not yet resolved.
• onCatch: A function to be called when an error is thrown. It will be called with the error as the first argument. If not provided, the error will be thrown instead.
• onPackExecution: A function to be called when a pack is executed. It will be called with the array of executed tasks (Promises yet to be resolved or rejected) as the first argument.

Considerations about chunk size

As of right now, if a chunk size is provided but not reached, the packer will not execute the tasks in the pack.

This means that if you provide a chunk size of 3 and you add 2 tasks to the packer, the packer will not execute the tasks until you add another task to the packer.

Please, keep this in mind when using the chunk execution method.

Tests

You can run the tests by using the following command:

npm test

How does it work?

Glossary

task

An async function to execute.

pack

The pack is the current list of async tasks to be / being executed.

queue

A temporary queue used to store every async task provided.

Execution Steps

The library exposes a higher-order function that you can wrap your async functions with to put them in a temporary queue.

The library will then move items from the queue to the pack based on the provided execution method (size limited for chunk configuration or time-based for interval configuration).

The library will execute the tasks in the queue according to the configuration that you pass to it.