Visual Regression Testing with Cypress 10

In this week’s post join me as I battle with some visual regression plugins for Cypress component testing to find the one which actually works. Spoiler alert - it’s not perfect either.

Blown away by the Beta release of Cypress component testing (as part of Cypress 10) I wondered if it would be possible to also have a component visual regression test.

I started by checking out what Cypress docs have to offer. The docs tell that Cypress does not include this ability as a feature, but since you can write custom plugins for Cypress, one can write a plugin which does just that - compares image snapshots.

One plugin which Cypress suggests is the cypress-plugin-snapshots, but if you wander into Cypress’ plugin “market” you can see that there are more.
After checking a few, I have decided to go with the cypress-image-diff-js plugin because it’s free, has a relatively simple api, seems to have a good update activity and a fair amount of GitHub stars.

I will attempt to introduce this visual regression testing to my Pagination component from the @pedalboard/components package. So here we go.

---

I already have Cypress component testing installed and enabled on my repo as described here, so let’s start with adding the plugin to my dependencies:

yarn add cypress-image-diff-js -D

Following their “Cypress integration” docs I first need to add the plugin to Cypress, but the example given is not compatible with Cypress 10. Let’s change that so it comply with it - on our cypress.config.ts we add it like so (see the “setupNodeEvents”):

import {defineConfig} from 'cypress';
import getCompareSnapshotsPlugin from 'cypress-image-diff-js/dist/plugin';

export default defineConfig({
   component: {
       devServer: {
           framework: 'react',
           bundler: 'webpack',
       },
       setupNodeEvents(on, config) {
           getCompareSnapshotsPlugin(on, config);
       },
   },
});

Now we need to register the new command to Cypress so it can be called from the tests. Since the command configuration was also changed in version 10, we need to add these lines in the cypress/support/commands.ts file:

import compareSnapshotCommand from 'cypress-image-diff-js/dist/command';
compareSnapshotCommand();

Just to make sure we didn’t break anything, let’s try and launch Cypress… yeah, it appears to be running as before. Now it is time to see if we can add the visual regression test to our existing test.

Here is how my test looks like now - the line with compareSnapshot is the one I’ve added in order to take the snapshot of the component. The string is the name of the snapshot and the number next to it is the tolerance threshold, which is currently zero:

describe('Pagination component', () => {
   describe('PREV button', () => {
       it('should be disabled when reaching the first page', () => {
           mount(<Simple />);

           const prevButton = cy.get('button').contains('PREV');

           prevButton.click();
           prevButton.click();
           prevButton.click();

           prevButton.should('be.disabled');

           cy.compareSnapshot('Pagination', 0.0);
       });
   });
});

I run the test and sure enough a new snapshot appears under the baseline directory. We also see that we have a comparison and diff directories currently empty, and a visual-report directory dir as well:

Here is my component snapshot:

These sorts of images, the baselines, are the ones that you need to commit into your source control, since the comparison is always done between them and the new generated ones.

Ok, now for the real thing - I’m going to present a brutal visual change to the component and see if the snapshots detected that as a visual regression.
I define that the disabled button will be in “aqua” color. I will make the change in the component’s CSS file and check the snapshots afterwards -

The test runs again but now it fails with this message:

Error
Image difference greater than threshold: 0

You can obviously change the tolerance threshold (even for each test separately, which gives a lot of flexibility) but there is no doubt here that the component suffers a major regression.

How do our snapshots look now?
Now I can see the additional images in the comparison and diff directories. The interesting one is the one in the diff directory:

Yes, we can clearly see the issue here. Anything in the reports dir?
Hmm…. nothing. Weird 🤔

Let’s say I approve this change, how do I update the snapshot?
I can update ALL the snapshots with the CLI command `cypress-image-diff -u’, but this is not the best option to go with. I think that a better way is to just delete the baseline snapshots you would like to update, and the plugin will take care of the rest.

Wrapping up

It’s nice, it’s free but… it’s still not quite there.
While other plugins fail to work completely in Cypress 10, this one works but opens the browser in a high resolution (at least on my machine) and not having an easy way to update selected snapshots is disappointing.
Having said that, it might be a good solution for those who do not want to pay extra money for 3rd party vendors which charge by the snapshot (cough, Chromatic, cough).

As always if you have any suggestion on how to make this better, comments or questions - be sure to leave them in the comments below :)

Hey! If you liked what you've just read check out @mattibarzeev on Twitter 🍻

Photo by Jason Dent on Unsplash

Comments

[Harshit Paul]

Cool read, quite informative, was wondering if there could have been basic content for Visual Regression Testing. Overall a great read!

Also sharing this basic tutorial on Visual Regression Testing

[Matti Bar-Zeev]

Cheers 🍻

[tausif29]

Hello @mbarzeev

I am using cypress with cucumber hence I use multiple-cucumber-html-reporter , When there is difference between actual and expected image, it just shows the error like Image difference greater than threshold: 0.2 but doesn't attach the screenshot with the report.

I don't want to use inbuilt report Cypress Image Diff Report as having 2 different reports doesn't make sense.

Is there anyway we can attach the failing screenshot with multiple-cucumber-html-reporter ?

[Matti Bar-Zeev]

Hi @tausif29,
Unfortunatly I don't have experience in integrating Cypress with Cucumber. I suggest addressing this issue to the Cypress team. If you find a solution, please share it :)

[Faith Berroya]

Hello sir, do you have one that uses Javascript? I tried github.com/meinaart/cypress-plugin... but is not running when I add import 'cypress-plugin-snapshots/commands'; in e2e.js.

Here is my issue in SO

(Edit)

Hello sir. I tried this plugin as well but I keep on getting this error although I have the folder on my directory. I'm currently stuck.

I added a ticket on their repository github.com/uktrade/cypress-image-d...

[Matti Bar-Zeev]

I will have a look on the issue you've opened and see if I can help somehow. Cheers.

[Faith Berroya]

Thank you sir!

[Matti Bar-Zeev]

I'm assuming that the .png file cannot be found under the comparison directory, the question is why...
Note that my cypress configuration adds the plugin for "component" testing and not for "e2e". I don't know if that difference makes the change, but keep in mind, perhaps there is a different configuration for "e2e"...
For the lack of a better solution ATM, I would try to run a "component" testing to see if that generates the required images and diff, and continue from there.

[Eugene Konneh]

Typical regression testing would require us to run comparison or visual testing on several pages or components.

How do you create multiple base images, one per Page or element under test?

[Matti Bar-Zeev]

In think it really depends on what you wish to test. You can have a page which holds amny components and test it as a bulk, or you can have a page for each component and be more detailed about it.
Tools like Chrimatic which have a nice Storybook integration allow you to have a visual regression test for each story and its oermutations.

[tausif29]

@faithberroya If you are using cypress with cucumber, update this in your cypress.config.js

await addCucumberPreprocessorPlugin(on, config, {
omitAfterScreenshotHandler: true,
});