Implementing an email queue with bullJs and hosting on Heroku!

#heroku #bull #node #redis

The problem...

If you've ever picked up a task that reads like this Setup an email service with SendGrid and nodemailer you probably did what most of us would do... Setup SendGrid for production and nodemailer for development and test environments. If you did that successfully and stopped no one would fault you but oftentimes email sending in some applications require just a bit more to keep em running smoothly. Few questions we often forget to ask is

How does the increase in volume affect my email setup?
Would it hold out or work when we have 10,000 users, sending at least one email every minute?
What happens when my email fails to send on the first attempt for various reasons...

Well, in this article we'll be looking at setting up an email queuing system using bullJs in a nodeJs application and also getting this setup to a production-like environment using Heroku.

Prerequisite

Basic knowledge of Javascript and nodeJs
Heroku account(you can sign up here)
Clone this repo 👉🏽 https://github.com/vic3king/bulljs-email-setup.git and follow the readme setup instructions.
Redis server running on localhost:6379. You can set one up quickly by following the instructions here

You can also set up your own nodeJs application the way you like it and integrate the code from this example and it should work regardless. But for this article, I will be working with the cloned repo.

The cloned repo contains some boilerplate code set up with node, express, and Sequelize. I added a registration endpoint that requires the user to input their username and email address(server/controllers/auth) to register. If you followed the setup instructions on the readMe you should be able to hit the registration endpoint at http://127.0.0.1:3333/v1/auth/register and register a user.

I have SendGrid and nodemailer preconfigured on the application as well in form of a function we can call whenever we need to send an email (server/config/emailSetup.js), I also added a base email template function that takes in a username, email, and a link as arguments. What we want to do now is to make sure that when a user registers they get an email that will be queued and processed by the bullJs package.

Bull is a Node library that implements a fast and robust queue system based on redis.

Although it is possible to implement queues directly using Redis commands, this library provides an API that takes care of all the low-level details and enriches Redis' basic functionality so that more complex use-cases can be handled easily. Read more

Bull Implementation

Navigate to the server/config folder and add a file called bullConfig.js.Paste in the following code.

import Queue from 'bull';
import dotenv from 'dotenv';

dotenv.config();

const { REDIS_URL } = process.env;

// Initiating the Queue with a redis instance
const sendMailQueue = new Queue('sendMail', REDIS_URL);

export default Sendmail queue;

Make sure your .env file contains REDIS_URL='redis://127.0.0.1:6379'. In this file, we're importing Queue from our package bull. with that, we create a new instance of the bull queue class and called it sendMailQueue. We could create multiple queues for a different use case like

const imageQueue = new Queue('image transcoding');

Make sure you have a Redis server running in the background

Next, we need to create a notifications.js file in the server/helper folder. In this file, we'll introduce two key concepts from bullJs producer and consumer. we'll also create a function that takes in a receiver's email address, name, and a link to verify their email address. we could extend this implementation by adding functions like a forgotPassword function that takes the same parameters as the registrationEmail function. for each new notification we send out, we need to trigger the producer by calling sendMailQueue.add(data, options);.

import emailService from '../config/emailSetup';
import sendMailQueue from '../config/bullConfig';
import template from './template';

/**
 *
 * @param {*} emailTo
 * @param {*} link
 * @param {*} name
 * @returns {*} sends an email to a new user
 */
const registrationEmail = (emailTo, link, name) => {
  const subject = 'Welcome to Karneek';
  const body = `<p>Dear ${name},</p>
  <p>We are thrilled to have you.</p>
  <p>Some random message with link</p>
      <a href="${link}" class="button">Confirm email</a>`;
  const message = template(subject, body, emailTo);

  const options = {
    attempts: 2,
  };
  const data = { emailTo, subject, message };

  // Producer: adds jobs to que, in this case emails to be sent out upon signup
  sendMailQueue.add(data, options);
};

// Consumer: this gets called each time the producer receives a new email.
sendMailQueue.process(async job => {
  emailService.mailSender(job.data);
});

const Notifications = { registrationEmail };

export default Notifications;

Finally, we need to call the registrationEmail function in our registration route. Navigate to server/controllers/auth.js. Import our notifications file

import notifications from '../helpers/notifications';

In the registerUser function, just after the verificatioToken variable add

      const { REGISTRATION_URL } = process.env;
      const verificationLink = `${REGISTRATION_URL}?token=${verificationToken}`;
      await notifications.registrationEmail(email, verificationLink, username);

That's it. To test things our run the server npm run start:dev and create a new user! On your console, you should see

Hosting this setup on Heroku

The challenge with this setup comes when we need to host this project on a remote server like Heroku. To get this working, we need to have an instance of Redis running by the side on Heroku. First, we need to add an addon by Heroku called Heroku Redis, we also need to add Heroku Postgres because this project uses a Postgres DB.

Navigate to your dashboard on Heroku, and create a new app. After that navigate to the overview tab of the newly created Heroku app and click on Configure Add-ons

This would navigate to a new page with a search box. Search for and add Heroku Redis, and Heroku Postgres

Copy the name of your Heroku app and on your terminal run the following commands.

heroku git:remote -a <name of heroku app>
heroku addons | grep heroku-redis
heroku addons:create heroku-redis:hobby-dev -a <name of heroku app>

After the last command, you should see something close to this

You need to copy your new Redis addon name at this point. for me, it was redis-deep-25660 (see screenshot)

run the next commands

heroku addons:info <your addon name> 
heroku config | grep REDIS
git push heroku master

If you're seeing anything like this on your logs, then you're done! 😳

To learn more about how Heroku Redis works ... https://devcenter.heroku.com/articles/heroku-redis#provisioning-the-add-on

Conclusion

In this article, we've been able to use bullJs for email queueing and also host our tiny app on Heroku by provisioning and running a Redis server on our Heroku app. I hope you found this useful, If you run into any problems you can check out the complete code here or leave a comment.