DEV Community

Need recommendations in Mocking database / Testing Cleanup (Unit Testing)

ｅｎｄａｎ on February 03, 2018

Hi everyone! I just started learning writing tests. I'm using Express, Mongoose, Mocha, Chai, and Chai-http. As of now, I'm just clearing my model...

Read full post

rhymes • Feb 3 '18

I'm going to be counter intuitive: don't mock your database, don't use any other database systems than the one you're going to use in production.

If you mock your database you're probably going to find out about actual problems with how you structured your models and data only in production which is not great :D

You can maybe use fixtures to prepopulate the test DB to speed up things a little bit but each test should run in isolation. As Alex said the testing time will probably increase a little bit but it shouldn't that much because you're not using a RDBMS.

So, populate the db, run the test, clean the db, repeat.

ｅｎｄａｎ • Feb 3 '18

That makes sense. So mocking is not required after all? Thanks!

rhymes • Feb 3 '18 • Edited

I would start thinking about mocking parts of the database only if the tests are extremely slow :-)

Bobby Priambodo • Feb 4 '18 • Edited

Let me be straight with you: as long as you're hitting the database, you're not doing unit testing.

But guess what, it's perfectly fine!

What you are doing right now is basically some form of end-to-end (i.e. integration) test. The "unit" under test is the app, and you test it by simulating requests to it. Might as well do it thoroughly and don't mock anything. You can also treat it like smoke testing.

Taking Michael Feather's words (the author of Working Effectively With Legacy Code---which, by the way, I highly recommend):

A test is not a unit test if:

It talks to the database

It communicates across the network

It touches the file system

It can't run at the same time as any of your other unit tests

You have to do special things to your environment (such as editing config files) to run it.

Tests that do these things aren't bad. Often they are worth writing, and they can be written in a unit test harness. However, it is important to be able to separate them from true unit tests so that we can keep a set of tests that we can run fast whenever we make our changes.

If, however, you want to do unit tests, then here's my suggestions when dealing with Express apps (or any other web API apps, in fact):

Separate your database access code from route handlers into a data-access layer.
Create integration tests for your database access code.
Create unit tests for your route handlers.

As a concrete example, I'm going to build a tightly-coupled implementation of a /v1/users route:

const Users = mongoose.model('Users')

app.get('/v1/users', (req, res) => {
  if (!req.isAdmin()) {
    // Do some unauthorized handling
  }

  Users.find((err, users) => {
    if (err) {
      // Do some error handling
    }

    res.json(users)
  })
})

In this case, you have no choice but to end-to-end test or to dirtily mock the mongoose or some other hacky way. Now, let's make it testable. First, we'll create the data access layer:

// ----- Users.js
// Data access layer. We allow database code here.
const mongoose = require('mongoose')
const UserModel = mongoose.model('Users')

const Users = {}

// Only expose functions that your app needs. Treat it as the
// public interface of your module.
Users.getAll = (cb) => {
  // Here we just pass the callback. In practice
  // there might be some logic involved, or if we
  // want to return a different shape of data than
  // what's on the database, we can modify them here.
  UserModel.find(cb)
}

module.exports = Users

In this file we allow database access. How do we test it? Well, by actually calling it and verify the database condition! This, again, is not going to be a unit test, but you shouldn't unit test your database access code anyway.

After that, let's define the handler!

// ----- getUsersHandler.js
// This is the request handler.
module.exports = (Users) => (req, cb) => {
  if (!req.isAdmin()) {
    // We return an object instead of calling res, since
    // it would make assertions easier.
    cb(null, { status: 401, responseJson: { unauthorized: true } })
  }

  // Here, we call the data access layer.
  Users.getAll((err, users) => {
    if (err) {
      return cb(err)
    }

    cb(null, { status: 200, responseJson: { users: users })
  })
};

Note that we don't require any module; not even Users! Instead, we pass Users as a parameter. Some may call this dependency injection, and that is exactly it. It will ease testing, which we will show later. With this, there are no tight-coupled dependency in the handler. It's only pure logic.

And then, to wire up, we have this, say in our app.js:

// ----- app.js
const app = ...
const Users = require('./Users')

// We inject the "real" Users module here!
const getUsersHandler = require('./getUsersHandler')(Users) 

app.get('/v1/users', (req, res) => {
  getUsersHandler(req, (err, result) => {
    if (err) {
      res.status(500)
      return res.json({ message: err.message })
    }

    res.status(result.status)
    return res.json(result.responseJson)
  })
})

Now, how to test the handler? Since it's just a function, well, we test it as usual:

const getUsersHandler = require('./getUsersHandler')

describe('getUsersHandler', () => {
  it('returns 401 when is not admin', (done) => {
    // We can even create a mock request object!
    const req = { isAdmin = () => false }

    // Create the handler, passing null as Users
    // (can be anything, really, since it shouldn't be called.)
    const handleRequest = getUsersHandler(null)

    // Call the handler!
    handleRequest(req, (err, result) => {
      if (err) return done(err);

      // Do assertions!
      result.status.should.be(401)
      result.responseJson.unauthorized.should.be(true)
      done()
    })
  })

  it('returns a list of users when Users.getAll is successful', (done) => {
    // This time, isAdmin should return true...
    const req = { isAdmin = () => true }

    // ...and we simulate Users.getAll successful call.
    // Create a dummy Users! We can define any kind of behavior that we want
    // just by creating an object that adheres to Users' interface.
    const userList = [{ id: 1 }, { id: 2 }] // etc.
    const Users = {
      getAll: (cb) => { cb(null, userList) }
    }

    // Create the handler, passing the dummy Users
    const handleRequest = getUsersHandler(Users)

    // Call the handler!
    handleRequest(req, (err, result) => {
      if (err) return done(err);

      // Do assertions!
      result.status.should.be(200)
      result.responseJson.users.should.be.an('array')
      done()
    })
  })

  // ...more tests to verify getUsersHandler behavior...
})

That's it! It's more ceremonious indeed, we have 3 files instead of the initial 1, but sometimes it's what you need to create a decoupled, testable code. There are no mocking frameworks, we only created fake objects, which are cheap and easy to understand and modify.

Do note that I wrote this code in dev.to's editor in one sitting and have not tested it at all :p so there might be mistakes, but I hope the idea is clearly delivered.

Would be glad to answer questions and hear criticisms :D

ｅｎｄａｎ • Feb 4 '18

Heyyy Bobby! First of all, thank you for the effort in writing all these!

Are you, by chance, what they call a purist? (Just kidding) Anyhow, I get your point in unit tests, it just makes sense. I gotta be completely honest with you that I do not (yet) completely understand the pseudocode you gave me. First, it uses ES6 syntax, second it looks so high-level/abstract-y way.

I am familiar with dependency injection in PHP but have not yet practised it yet in Node. Also, it's the first time I have heard of data-access layer. I think I might be re-reading your answer for a few times before I can say something useful.

But really, thank you for this! I'll take my time digesting your comment. 😊

Bobby Priambodo • Feb 4 '18 • Edited

Oh, I just realized we've interacted before in a comment thread. Nice to see you again, Endan (if that's your name)!

I'm a purist in the sense that I love pure functions :p (because they're easy to test!)

I planned to write a blog on this topic since a long time ago, but haven't quite had the chance. Writing this comment somehow brought up some concepts that I thought to put in the blog, so it came out longer than I initially expected.

Take your time! I tried to keep it simple, but it might very well be my bias showing. Feel free to ask more questions. Good luck on your testing journey :)

amehmeto • Jul 21 '20

Very helpful comment. I am begging you to write posts about those knowledge 🙏

Mark Yang • Jan 17 '19

Sounds like a great solution. Cheers!

Alex Rudenko • Feb 3 '18

For such API level tests, I usually use a real database instance running locally, and I clean up the entire database before every test and insert objects required for a test case using the models (in this case, it would be mongoose models). Pros: it gives more confidence that things run fine. Cons: 1) the time needed to run tests increases. 2) you need to set-up a test DB

Isaac Lyman • Feb 3 '18

I think what you're already doing is pretty much the way to go. You might want to be a little more specific about which user you're deleting, but that's the only change I would make.

If you want an example of an Express app that runs integration tests using a real database, here's one I've been working with:

github.com/isaaclyman/Edward-the-A...

This is the test utilities file, which is imported by most of the test suites.

Before every test, I run "deleteTestUser", which deletes all the content in the database that is foreign-keyed to the test user. Then I run "createTestUser", which re-creates the test user. And then I have utility methods for creating content for the test user.

I'm using Ava/supertest/Postgres/knex, so the tech is different, but the principle is the same.

One thing I recommend you keep doing is to do all test cleanup in the beforeEach section of the test suite. If you do it in afterEach and the test fails, the data won't be cleaned up, and your next test run will be polluted with old test data.

Dian Fay • Feb 3 '18

Mocking your data is useful if you have a rich domain layer (models with their own suites of functionality above and beyond CRUD), or as rhymes said, if your tests are too slow with real data (which often happens with rich domain layers).

There are better and worse ways to approach populating the database for real, though. You don't want to have each testcase initialize and clean up its own fixture data: if something in your data model changes, you have to correct every single test that touches on it. And teardown is different for each testcase, leading to difficult-to-diagnose issues with data pollution.

Your data model is fundamentally a dependency graph. You can't, for example, have prescriptions without doctors to issue them and patients to whom they're issued. All of your test cases operate on subgraphs of your model: doctors do many other things besides write prescriptions, after all.

The most flexible way I've found to approach fixture data is to compose reproducible datasets. You're testing each functional unit in your data model individually and in combination with other units. So if you have some code which creates a couple of doctors, some code which creates a few patients, and some code which creates some prescriptions linking the previous in various combinations, you can apply as many of those pieces as you need for a given testcase in the before or other setup method. And you can have a single cleanup function which removes every possible record the fixtures could generate. All your fixture code stays in one place and tests consume what they need.