Securing your component library with visual testing

Find out how we added visual testing to our component library (Homeday-blocks) and why I think it is important to have visual testing in your test toolkit.

Background

As an introduction I want to share how I started with visual testing and why I decided to add it to our component library.

Currently we’re working on a new Homeday’s project and I decided to enhance our unit tests with some integration and visual tests. The idea is to have a secure and robust project, in which we feel comfortable adding new features, refactoring code and fixing future issues — which hopefully we will not have 😅

We’re using Cypress in this project, so I decided to integrate screenshot comparison tool with it and see how that works together. Unfortunately it didn’t work that well. I’ve tried two tools: cypress-plugin-snapshots and cypress-image-snapshot. Both are great, as they are free, however when dealing with visual testing you have to be ready to adjust some parameters and find magic numbers that work for you. As it is not my main focus to really dig deep into how visual testing really works (maybe in the future), I decided to find a better and mature tool to do so. I ended up using Percy. Just as a disclaimer: I’m not being sponsored by Percy.

Reading through Percy docs, I saw that they have an easy solution to integrate with Storybook. The good news is that our component library already uses Storybook. The bad news is that not all the components have stories (which means they don’t appear in Storybook). Since we introduced unit tests in our component library I’ve been thinking about ways to expand it and make our component library more robust then it currently is. That’s why I ended up trying out visual testing.

Every quarter we have a week to develop and try new stuff in the company. It is called PEW (Product Engineering Week). It works as following: Week before you select your topic; Monday morning you do a quick intro to your topic; Friday afternoon you present your results. During this week all ours meetings are canceled so we can really focus on our topics. My PEW project this quarter is to add visual testing to our component library =]

First steps

As I mentioned before, I’m going to use Percy. In order to do that you need an account and add your project there. It just asks you to authorize access to your repo account, that way you can see the status of the visual testing in your PR. This is not mandatory, but it helps you to keep track of your tests, specially if you’re opening a PR or so. I’m also relying on Storybook as our component library already uses it.

Most of what I'm going to show here can be found in Percy docs for Storybook as well.

You just need one dependency to start using Percy + Storybook: @percy/storybook 😅

npm i --save-dev --save-exact @percy/storybook # --save-exact is optional

After installing the dependency we need a PERCY_TOKEN environment variable to secure and match the local project and the Percy project. I also recommend to add it to your CI, as it is probably the tool that will fire your tests.

export PERCY_TOKEN=<my secret token here>

Before we run Percy, we need to first build Storybook, otherwise Percy doesn’t know where to find the stories. Now we can simply run npx percy-storybook command and it is going to screenshot every single story from Storybook.

With this simple setup, we can already benefit from the powers of visual testing. It is even better if you’re just starting a project, so you can carefully cover each case you want with visual testing. As we already have an in-use, project we can’t do that. We needed some further adoptions that I'll describe in the following section.

Knobs, repetitive stories and other corner cases

Our component library is open source, so I’ve also opened the Percy builds, so everyone that is interested can follow it as well.

The first issue I encountered was related to our Homepage. After running the first test I saw it took a screenshot from it.

We have a Welcome component which renders the first page in Storybook, so it becomes our Home. I don’t want to visual test it, so I had to find a way to ignore it. Percy docs are pretty clear about that, you simply add skip parameter to your story and it gets ignored.

storiesOf('Welcome', module)
  .addParameters({ percy: { skip: true } })
  .add('Welcome', () => ({
    ...

Second issue was similar to the first one, but the scenario is a bit different. We have some components with knobs, which allow the users from Storybook to play with them. It basically binds the component’s props with some inputs, so you can nicely interact with the component. I don’t want to test those. I need to create all possible variations in stories so I can take screenshots from all of them and compare over time. The knobs are great for users as they can play with the components, but not that useful when dealing with visual testing. In order to avoid that, I had to skip some Playground Stories (that is how we call the stories with knobs) and refactor some stories as well.

stories
  .add('Playground 🎛', () => ({
    components: { HdButton },
    props: {
      ...
    },
    template: `
      <HdButton
        :modifier=modifier
      >{{ text }}</HdButton>
    `,
  }), { percy: { skip: true } });

One of our stories makes sense only on mobile view, as it adapts the component on mobile. This is our third issue, which can be solved simply by adjusting the configuration. Instead of taking screenshots on mobile and desktop, I’ve added a parameter to specify the screen size. Just as a reminder: Every extra screenshot will add to your account limit and also will take extra time to process, so I decided to really avoid the unnecessary screenshots from the beginning.

}), { percy: { widths: [375] } });

The last issue: CI integration. As the project is open source, it means that people can fork it and open a PR. We (developers from the company) follow the same approach. We have our forks and open PRs from them to the main repo. We use Travis as CI, but it happens on other CIs as well, it doesn’t share environment variables to fork builds, due security issues. It means that PERCY_TOKEN won’t be available when CI builds. Without PERCY_TOKEN, the Percy command fails, which also fails the CI build. To avoid that, I had to add a check in the test command for the existence of the PERCY_TOKEN. If it does not exist, we skip the tests. If you want to run Percy in this scenario, you have to checkout the desired branch and run tests locally, with PERCY_TOKEN exported. Alternatively, you can create a new branch in the origin repo that will trigger the build.

"test:percy": "if [ $PERCY_TOKEN ]; then percy-storybook; else echo 'Skipping test: PERCY_TOKEN is not defined'; fi"

Final thoughts

You can find the final PR here. Fell free to add your comments and questions there.

The integration Percy + Storybook works smoothly and it is relatively simple to add it to a project. Percy also offers a pretty good free tier so you don’t have to worry about investing some money from the beginning.

There are still some challenges to be tackled, especially components that request some interaction or event to happen. Those need to be better explored and tested. It will probably lead to new stories being written.

Percy also allows you to customize your integration even deeper if you want, I do suggest you to read its documentation if you need to do such a thing.

I would recommend you to give it a try and also share your learnings and sorrows [=

Happy coding \o/

Thanks to Ilyas, Ante and Sinisa for reviewing the text.