[SCARY] Visual Regression Testing

Background

During one of our releases, we had to roll all the sites back (5 of them) because one of the developers found that a single page had changed unexpectedly. The rollback took almost 4-hours.

I kept thinking that we found this issue on one of the sites. There was no guarantee that the issue was on all of the sites.

Solution

I wanted something that could store information on a group of pages prior to the release and compare them against the pages after the release.

After checking with some friends, I looked into Playwright, a library generally used for end-to-end testing.

I started with this guide: Visual Comparisons.

The Code

I write a lot of Unit, Component, and Integration Tests.

I broke all the rules.

At it's core, I want to be able to test over 1-8 domains; quickly turning them off and on, as needed. Additionally, for each domain, I can have a lengthy list of paths to test.

I came up with something like this as a configuration file: /tests/smoke-tests.json.

{
  "SMOKE_ONLY": true,
  "MAX_DIFF_PIXEL_RATIO": 0.001,

  "DOMAINS": [
    { "url": "https://www.bobfornal1.com", "active": false },
    { "url": "https://www.bobfornal2.com", "active": false },
    { "url": "https://www.bobfornal3.com", "active": false },
    { "url": "https://www.bobfornal4.com", "active": true },
    { "url": "https://www.bobfornal5.com", "active": true },
    { "url": "https://www.bobfornal6.com", "active": true },
    { "url": "https://www.bobfornal7.com", "active": true },
    { "url": "https://www.bobfornal8.com", "active": true }
  ],

  "https://www.bobfornal4.com":  {
    "smoke": [
      "/",
      "/discover/gd/",
      "/discover/sa/eavestroughs/",
      "/discover/sa/gutter-guard/",
      "/discover/sa/installation/"
    ],
    "detailed": []
  }
}

The smoke and detailed keys give me the option to store paths that aren't being used in a way that they aren't tested, but could be quickly with the SMOKE_ONLY option.

MAX_DIFF_PIXEL_RATIO is how closely we want to look a differences. The closer to zero, the more accurate a match must be.

My playwright.config.ts file looks like this.

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',
  use: {
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },

    {
      name: 'Mobile Chrome',
      use: { ...devices['Pixel 5'] },
    },
  ]
});

Clearly, more browsers could be used, but this is enough for my use-case.

Here's the test code: /tests/smoke-tests.spec.ts.

import { test, expect } from '@playwright/test';
import core from './smoke-tests.json';

interface DomainPattern {
  url: string;
  active: boolean;
}

const smokeOnly: boolean = core.SMOKE_ONLY;
const maxDiffPixelRatio = core.MAX_DIFF_PIXEL_RATIO;

test.describe('Smoke Tests', () => {

  core.DOMAINS.forEach((domain: DomainPattern) => {
    if (domain.active === true) {

      const paths: Array<string> = smokeOnly
        ? [...core[domain.url].smoke]
        : [...core[domain.url].detailed, ...core[domain.url].smoke];

      paths.forEach(async (path: string) => {
        const fullpath: string = `${domain.url}${path}`;

        test(`for "${fullpath}"`, async ({ page }) => {
          await page.goto(fullpath, { timeout: 60000, waitUntil: "networkidle" });
          await page.waitForTimeout(5000);
          await expect(page).toHaveScreenshot({
            fullPage: true,
            maxDiffPixelRatio,
          });  
        });

      });

    }
  });
});

What It Does

This code looks through the domains, builds an array of paths and loops through those. Each domain path combination is a test (note the name change to "for ${fullpath}" ensuring a different name for each test.

We go to the page, wait 5 seconds, and expect a screenshot comparison of the full page.

Tooling

This code is run from the command line with

npx playwright test

Simple.

You then get something like this in the command line.

... and, your browser opens with the test results.

I generally click "Failed" to reduce what I need to check.

At this point, I need to manually check the pages. Clicking on one of the failed tests takes me to a page; scrolling down I can see a heatmap overlay (note the red).

Clicking the "Slider" option is one of my favorites that allows us to compare the before and after changes by sliding a scrollbar left and right.

Remember

REMEMBER that this test suite has to be run before and after each release. The before run generates the new images, so you should delete the created /tests/smoke-tests.spec.ts-snapshots folder each time to allow for new image captures.

Summary

Scary?

No. This test is 37-lines long and provides a level of assurance that got applause the first time it was run for this team.

I am able to test almost 300 pages in a matter of minutes; that's powerful!