Sharing a Redis using a Prefix

Using Bull Prefix to Isolate Jobs Between Developers

When developers share resources from staging to develop locally, issues can arise, particularly when using shared Redis instances and Bull for distributed jobs. For instance, one developer might unintentionally consume jobs created by another, leading to conflicts and confusion.

To learn more about the benefits of leveraging staging services in development, check out this detailed guide: Enhancing DX by Using Staging Services in Development.

One effective solution to this problem is using job prefixing. This strategy allows each developer to have their namespace or prefix for jobs, ensuring that jobs created by one developer are only processed by their workers. Here's how you can implement this:

Why Use Job Prefixes?

• Prevent Job Conflicts: Multiple developers may add jobs to the same queue without realizing it, leading to a mix-up of jobs.
• Isolation of Developer Work: Each developer can have a unique prefix that only their jobs can be consumed by their workers.
• Scalability: As the project grows and more developers work with queues, job prefixes help keep things organized.
• Cost-Effective: Eliminates the need to spin up a new Redis instance for every developer.

Using Bull Prefixing for Job Isolation

Bull, a popular queue system for Node.js, supports prefixes for queue and job names, allowing you to easily create isolated queues for each developer.

By leveraging this feature, you can ensure that developers working in parallel don’t interfere with one another. Here’s how you can implement this approach:

Enabling Prefixing with BULL_DEV_PREFIX

Using process.env as feature flags is a common practice for enabling or disabling experimental or development-only features. If you're unfamiliar with this approach, check out this guide: Using process.env as Feature Flags

To simplify the configuration for developers, you can introduce an environment variable (BULL_DEV_PREFIX) in their local .env file. This prefix ensures that all jobs created by a developer will only be consumed by their workers. If no prefix is set, jobs will remain accessible to others, including staging.

This setup is particularly useful when testing new job implementations without affecting shared queues.

Example Implementation:

const getDevPrefix = () => {
  if (process.env.BULL_DEV_PREFIX) {
    return {
      prefix: process.env.BULL_DEV_PREFIX,
    };
  }

  return {};
};

const myQueue =new Queue('MY_QUEUE', config.REDIS_HOST, {
    ...getDevPrefix(),    
})

In this example:

The BULL_DEV_PREFIX variable is read from the environment.
If set, the prefix is applied to the queue, ensuring isolation.
If not set, the queue behaves normally, allowing staging and other developers to interact with it.

Diagram with before and after the solution

Conclusion

Using Bull prefixes is a simple and effective way to improve collaboration in a multi-developer environment. It ensures:

Developers can work independently on shared queues.
Conflicts are avoided.
Staging resources are utilized efficiently.
This approach boosts workflow efficiency and scalability, allowing teams to focus on development without stepping on each other’s toes.

---

Woovi
Woovi is a fintech platform revolutionizing how businesses and developers handle payments in Brazil. Built with a developer-first mindset, Woovi simplifies integration with instant payment methods like Pix, enabling companies to receive payments seamlessly and automate financial workflows.

If you want to work with us, we are hiring!

---

Photo by cottonbro studio