Don't use fixtures in Cypress and unit tests- use factories

Unit tests are great... when they work dependably! In fact, there's an old saying that "a bad test is worse than no test at all." I can attest that weeks spent chasing down a randomly "false negative" test is not efficient. Instead, you could have been using that time to write business code that helps the user.

So let's talk about one of these easiest techniques to write less brittle tests: test data factories.

But before we get into what factory functions are and why you would want to use them, let's first try to understand the type of brittle test that they eliminate.

Aspects of tests we want to avoid

1. tight coupling
2. lack of type safety (which causes lengthy refactoring and bugs)
3. giant fixture folders

Factory functions will fix all of that.

So what are factory functions?

A factory function is a function that creates an object. Simple as that. Yes, there is the "abstract factory" pattern popularized by the Gang Of Four's Design Pattern book decades ago. Let's make it nice and simple.

Let's make a function that makes it easy to make stuff so we can test more easily.

Here's the world's most simple example:

interface ISomeObj {
  percentage: string;
}

export const makeSomeObj = () => {
  return {
    percentage: Math.random()
  };
}

Let's see how such a simple pattern can be used to fix the aspects of brittle tests that we described above.

We'll start by describing how tests are typically written and then we'll evolve the solution iteratively as we solve each of the problems.

A Real World Example Of How Brittle Tests Occur

It all starts innocently. You or another motivated developer on the team wanted to pay it forward and add a unit test for one of the pages. To test the function you save some test data in a JSON file. Cypress (the most amazing UI testing library at the time of this writing) even encourages you to use a test data fixture JSON file. But the problem is... it's not even remotely type safe. So you could have a typo in your JSON and spend hours chasing down the issue.

To illustrate this let's look at example business code and test automation code. For most of these examples we'll assume that you work at an insurance company that explains how the rules work for each state within the United States.

// This file is "src/pages/newYorkInfo.tsx"
import * as React from 'react';

interface IUser {
    state: string;
    address: string;
    isAdmin: boolean;
    deleted: boolean | undefined;
}

export const NewYorkUserPage: React.FunctionComponent<{ user: IUser }> = props => {
    if (props.user.state === 'NY' && !props.user.deleted) {
        const welcomeMessage = `Welcome`;
        return <h1 id="ny-dashboard">{welcomeMessage}</h1>;
    } else {
        return <div>ACCESS DENIED</div>;
    }
};

The code looks good, so let's write some JSON to store the positive test case.

// fixtures/user.json
{
    state: 'NY',
    isAdmin: true,
    address: '55 Main St',
}

And now the test code. I'll demonstrate the issue using some psuedo-code for a Cypress test, but you can imagine this occurring any test code where you load the fixture and run your assertion.

// When the UI calls the user endpoint, return the JSON as the mocked return value
cy.route('GET', '/user/**', 'fixture:user.json');
cy.visit('/dashboard');
cy.get('#ny-dashboard').should('exist')

Looks fine, and it works perfectly until you need to test another scenario involving a different user. What do you do then?

Bad Solution - If one file worked, just keep making JSON files

Should you simply create another JSON fixture file? Sadly, this simple solution happens all the time because it's the easiest (at first). But as the number of cases grows, the number of JSON files grow too. You would need 52 different JSON files in order to test every page for every user in the United States. When you start testing if a user is or isn't an administrator, you would have to create 104 files. That's a lot of files!

But you still have the problem of type safety. Let's say the Product Owner comes to the team and says "I want to be kind and display the name of the user when we welcome them."

So you add the name property to the interface and update the UI to handle for this case.

// This file is "src/pages/newYorkInfo.tsx"
import * as React from 'react';

interface IUser {
    name: string;
    state: string;
    address: string;
    isAdmin: boolean;
    deleted: boolean | undefined;
}

export const NewYorkUserPage: React.FunctionComponent<{ user: IUser }> = props => {
    if (props.user.state === 'NY' && !props.user.deleted) {
        const welcomeMessage = `Welcome ${props.user.name.toLowerCase()}!`;
        return <h1 id="ny-dashboard">{welcomeMessage}</h1>;
    } else {
        return <div>ACCESS DENIED</div>;
    }
};

It's great that you updated the business code, but the fixture JSON is out of date. And because the fixture JSON doesn't have a name property, you get the following error:

Uncaught TypeError: Cannot read property 'toLowerCase' of undefined

Now you have to add the name property to all 52 user JSON fixture files. We can solve that with Typescript.

Slightly Better Solution - Move it into a TypeScript file

By moving the JSON out of the fixture file and into a .ts file, the Typescript compiler finds the bug for you:

// this file is "testData/users"
import {IUser} from 'src/pages/newYorkInfo';

// Property 'name' is missing in type '{ state: string; isAdmin: true; address: string; deleted: false; }' but required in type 'IUser'.ts(2741)
export const generalUser: IUser = {
    state: 'NY',
    isAdmin: true,
    address: '55 Main St',
    deleted: false,
};

And we'll update the test code to use this new object.

import { generalUser } from 'testData/users';

// When the UI calls the user endpoint, return the JSON as the mocked return value
cy.route('GET', '/user/**', generalUser);
cy.visit('/dashboard');
cy.get('#ny-dashboard').should('exist')

Thanks Typescript! As soon as you solve the compiler error by adding name: 'Bob Smith' into the generalUser object, the code compiles cleanly, and best of all... your test passes again!

You've met one of our three goals by achieving type safety. Unfortunately the tight-coupling problem still exists.

For example, what happens when a developer who is new to unit testing comes along. All they were thinking about is that they need to test a feature that involves a deleted user. So they add deleted: false to the generalUser object.

Kaboom! Your test fails and their test passes. That's what it means to be tightly-coupled.

So the developer spend a few minutes (or hours) debugging and they realize that both tests share the same setup data. So the developer uses the easy (but short-sighted solution) from before and they simply create another object deletedUser so that there's 1 object per test. This can get out of hand quickly-- I've seen test data files that are 5000 lines long.

Click here to see how insane this can be.

// this file is "testData/users"
import {IUser} from 'src/pages/newYorkInfo';

export const nonAdminUser: IUser = {
    name: 'Bob',
    state: 'NY',
    isAdmin: false,
    address: '55 Main St',
    deleted: false,
};

export const adminUser: IUser = {
    name: 'Bob',
    state: 'NY',
    isAdmin: true,
    address: '55 Main St',
    deleted: false,
};

export const deletedAdminUser: IUser = {
    name: 'Bob',
    state: 'NY',
    isAdmin: true,
    address: '55 Main St',
    deleted: true,
};

export const deletedNonAdmin: IUser = {
    name: 'Bob',
    state: 'NY',
    isAdmin: false,
    address: '55 Main St',
    deleted: true,
};

// and on and on and on again...

There has to be a better way.

Good Solution: Factory Function

So how do we refactor the giant file of objects? We make it one function!

// src/factories/user
import faker from 'faker';
import {IUser} from 'src/pages/newYorkInfo';

export const makeFakeUser = (): IUser => {
    return {
        name: faker.name.firstName() + ' ' + faker.name.lastName(),
        state: faker.address.stateAbbr(),
        isAdmin: faker.random.boolean(),
        address: faker.address.streetAddress(),
        deleted: faker.random.boolean(),
    }
}

Now every test can just call makeFakeUser() when they want to create a user.

And the best part of this is by making everything random within the factory, it clarifies that no individual test owns this function. If a test ones a special kind of IUser, they're going to have to modify it on their own later.

And that's easy to do. Let's imagine the deleted user test where we don't care what the user's name is or anything. We only care that they're deleted.

import { makeFakeUser } from 'src/factories/user';
import {IUser} from 'src/pages/newYorkInfo';

// Arrange
const randomUser = makeFakeUser();
const deletedUser: IUser = { ...randomUser, ...{
  deleted: true
};
cy.route('GET', '/user/**', deletedUser);

// Act
cy.visit('/dashboard');

// Assert
cy.find('ACCESS DENIED').should('exist')

For me, the beauty of this approach is that it's self-documenting. Anyone who is looking at this test code should understand that when the API returns a deleted user, we should find "Access Denied" on the page.

But I think we make this even cleaner.

Best Solution: easy overriding with mergePartially

It was acceptable to use the spread operator above since it was a small object. But this can be more annoying when it's a heavily nested object like this one:

interface IUser {
    userName: string;
    preferences: {
        lastUpdated?: Date;
        favoriteColor?: string;
        backupContact?: string;
        mailingAddress: {
            street: string;
            city: string;
            state: string;
            zipCode: string;
        }
     }
}

You really aren't going to want to have hundreds of those objects floating around.

So if we allow users to override only what they want, we can make for some really simple and DRY setup code. Imagine there's a very specific test that must have a user who lives on "Main Street."

const userOnMainSt = makeFakeUser({
    preferences: {
        mailingAddress: {
            street: 'Main Street'
        }
    }
});

Wow, they only needed to specify what they needed for the test instead of the other 7 properties. And we didn't have to store a one-off object in some giant test file. And we met our self-commenting goals as well.

And how do we enhance our makeFakeUser function to support this kind of partial override? Check out how easy the mergePartially library makes this (full disclosure: I am the mergePartially maintainer).

const makeFakeUser = (override?: NestedPartial<IDeepObj>): IDeepObj => {
        const seed: IDeepObj = {
          userName: 'Bob Smith',
          preferences: {
            mailingAddress: {
              street: faker.address.streetAddress(),
              city: faker.address.city(),
              state: faker.address.stateAbbr(),
              zipCode: faker.address.zipCode(),
            },
          },
        };
        return mergePartially.deep(seed, override);
      };

Let's see how clean the final test code looks. You'll notice that we saved multiple lines of code and our setup data is fresh every time:

import { makeFakeUser } from 'src/factories/user';
import {IUser} from 'src/pages/newYorkInfo';

// Arrange
const deletedUser= makeFakeUser({
  deleted: true;
});
cy.route('GET', '/user/**', deletedUser);

// Act
cy.visit('/dashboard');

// Assert
cy.find('ACCESS DENIED').should('exist')

Wrap Up

Thank you for reading along in the evolution of how we took our test code from brittle and huge test code to tiny and independent.

I'd love to hear from you on your thoughts of this approach.

Comments

[Eli Tamošauskas]

Loving every bit of this! Thanks Dan!

[Dan Greene]

Thanks you so much for the positive energy. It certainly has been a pattern that has worked well for the teams at our company in the time since we adopted the pattern.

[Armando Cifuentes González]

Just for curiosity.- To avoid this error Type 'string | undefined' is not assignable to type 'string'. Type 'undefined' is not assignable to type 'string'., without using your library, can I solve it by adding the ! at the end of the property I want to use (I mean, something like this: const field: string = faker().field!;)?

[Dan Greene]

To avoid that error you have to check to see that the variable is not undefined.

So

if(yourVariable === undefined){
  throw new Error (“an error message explaining why this is happening”);
}

[Armando Cifuentes González]

Ok!

[Ben Mitchinson]

Great write up, thanks for taking the time to describe the issue. Just ran into similar issues, and this provides clear fixes + is a great reference for my team.

[Dan Greene]

Thank you kindly Ben. And please ask them to share with others. I’d like to see this become a standard within the TS Cypress community.

Again, it’s so generous of you to say thank you. Isn’t dev.to such a wonderful community? :)

[Josh Schneider]

Very nice article, @dan

All they were thinking about is that they need to test a feature that involves a deleted user. So they add deleted: false to the generalUser object.

Is this supposed to read "So they add deleted: true..." or am I confused?

[Chim]

🔥🔥🔥

[Marijana Rukavina]

Thanks Dan for writing this article. It sounds like a direction I would like to go to. I have one question though. How would you solve for Typescript the fact that cypress ui tests need to run on different environments? Meaning you would have to create/load test data objects with different values for different environments. I would appreciate if you could share your thoughts about it. Thanks 😊

[Dan Greene]

My apologies for not seeing this question sooner. So I would first ask thy you would need different data for mocked tests. Generally mocked tests are designed to test the "shape" of your data, not necessarily real data. As where end-to-end (e2e) tests would rely on both the shape and the content of the data and would therefore not utilize mock data (or factories) at all.

But if you still choose to mock different data per environment, I would consider simply clarifying that in your tests.

let user: IUser | null = null;
if(process.env.NODE_ENV === 'production'){
  user = makeFakeUser({userName: 'John'});
} else if(process.env.NODE_ENV === 'staging'){
  user = makeFakeUser({userName: 'Susan'});
}
// Assert so that the user variable can no longer be seen as null
if(!user){
  throw new Error(`unable to initialize mock user since NODE_ENV was invalid`);
}
// the rest of your test