Visual Testing with Chromatic

Have you ever pushed a commit into prod, only to notice later it made your site look straight out of a Hieronymus Bosch panel? If so, we’ve got you covered! This article will explain how introducing visual regression testing with Chromatic into your workflow will help you avoid unintended UI changes.

What is visual testing?

When developing a user interface, there are two important targets for testing: the behavior of the graphical elements, and how they are presented and arranged. The former is usually achieved by unit and E2E tests, while for the latter it’s common to leverage snapshot tests. Snapshot tests work by comparing the output of a test against a version-controlled golden file, failing on a mismatch. Intentional changes include an update to that golden file.

Tools like Jest make it easy to create non-visual snapshots based on the markup of an interface. These tests are useful to alert PR authors that they may be introducing unintended changes, but it makes it hard on reviewers to validate and approve the intended ones: it’s not an easy task to mentally conjure up the exact visual changes from looking at HTML alone. Developers reviewing a changeset need to spin up two versions of the UI and manually track down the changes. If the UI under test can have many states and variations, this can easily turn into a long and laborious task: maybe it was the layout of an error message that changed, or the position of a spinner rendered when the component is loading data. In the context of web development, having a published, trunk-based Storybook is key to this workflow.

To make matters worse, markup snapshots don’t capture externally-defined styles, as is the case with HTML and CSS. This is where visual regression tools like Chromatic really shine, extending the UI snapshots to their complete, rendered state, and layering a review process on top. Figure 1 contains an example of the Chromatic review screen for a visual test.

Fig. 1 - Chromatic review screen for a snapshot

On the left we have the snapshot from a previous build and on the right the snapshot with the changes we’re introducing, highlighted in green. Reviewers can comment on each snapshot, accepting or rejecting the changes. Pretty great, right?

In the following sections we’ll cover how to create these snapshots and integrate Chromatic into a CI pipeline.

Writing snapshots

Chromatic integrates with Storybook, capturing a screenshot of each story on a configurable set of browsers (Chrome by default), for a given set of viewports.

At Umani, we like our Storybook stories to be interactive and expose a bunch of controls. Figure 2 contains an interactive story for an Avatar component.

Fig. 2 - An interactive story with multiple controls

This story is written as:

export default {
    title: "Avatar",
    parameters: {
        chromatic: {
            viewports: [360, breakpoints.desktop],
        },
    },
}

interface AvatarStoryProps extends AvatarProps {
    readonly showContent?: boolean
}

const Template: Story<AvatarStoryProps> = ({ showContent = false, size, ...args }) => {
    return (
        <Avatar size={size} {...args}>
            {showContent ? (
                <Stack space="xs">
                    <Text size="md">Art Vandelay</Text>
                    <Text size="sm" variation="subtle">
                        View profile
                    </Text>
                </Stack>
            ) : null}
        </Avatar>
    )
}

export const Basic = Template.bind({})

Basic.args = {
    showContent: false,
    size: "md",
}

Basic.argTypes = {
    showContent: {
        name: "Show Content",
        description: "Content is shown to the right.",
    },
    size: {
        name: "Size",
        description: "Avatar size.",
    },
}

These stories don’t make up a very good snapshot, which is why we disable Chromatic in their parameters:

Basic.parameters = {
    chromatic: {
        disabled: true,
    },
}

The stories we are interested in capturing with Chromatic visual tests are fine-grained and uninteractive. We’ll usually include (a sensible version of) the cartesian product of all variations of a given component within a snapshot. For example, the snapshot story for our Avatar component is defined as:

export const Snapshot: Story = () => {
    const stories: Story[] = []
    for (const showContent of [true, false]) {
        for (const size of ["sm", "md"] as const) {
            const props = { showContent, size }
            const story: Story = () => <Template {...props} fallback="" />
            story.storyName = `Avatar with photo, with${!showContent ? `out` : ``} content and size ${size}`
            stories.push(story)
        }
    }
    return <StoryGroup stories={stories} />
}

Figure 3 contains the rendered snapshot story.

Fig. 3 - A rendered snapshot story containing multiple variations

The reason we bundle different variations into the same story is so we don’t blow up our snapshot budget. Similarly, we’ll strive to minimize duplicate snapshots: if the variations of a component like Avatar have already been tested in isolation, we may not need to include them when using that component in a composite story. Minimizing stories is helpful to remain within limits and also curb the time it takes to review changes.

Notice that we configure Chromatic to produce two snapshots at two different viewports with

chromatic: {
    viewports: [360, breakpoints.desktop],
}

This is useful for responsive components and pages.

Snapshotting CSS states like hover and focus often require using Storybook play functions or the ability to trigger those states from component props.

Setting up Chromatic with Github Actions

At Umani we use Github Actions for our CI pipeline. Integrating Chromatic is very easy, but not without its subtleties. This is our workflow job which builds and publishes the Storybook into Chromatic:

    storybook:
        name: Storybook

        runs-on: ubuntu-latest

        steps:
            - name: Checkout
              uses: actions/checkout@v2
              with:
                  fetch-depth: 0
                  ref: ${{ github.event.pull_request.head.sha }}

            - uses: ./.github/actions/load-node-modules

            - name: Create snapshots
              run: yarn chromatic --only-changed --skip 'dependabot/**'
              env:
                  CHROMATIC_PROJECT_TOKEN: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
                  CHROMATIC_SHA: ${{ github.event.pull_request.head.sha }}
                  CHROMATIC_BRANCH: ${{ github.event.pull_request.head.ref }}

There’s a few things to unpack here, but the important bits are straightforward: ​​we check out the PR’s code (with full history, which is required by Chromatic), use a composite action to load the node modules from cache, and invoke Chromatic. (There’s an official Github Action, but we are not yet leveraging it.)

This job generates a unique build in Chromatic. A branch/PR can have many builds and, unless otherwise specified, snapshots are checked for differences against their counterparts from a previous build either on the same branch or belonging to an ancestor commit. Chromatic’s documentation goes into detail about how baselines are calculated. For us, that baseline is either a build within the same branch or a build for the main branch. Since we’re not using Chromatic’s UI Review tool and we squash our PRs, there’s no association between the merge commit and the commits on the merged PR. This means Chromatic can’t establish the builds of a merged PR as the baseline for new PRs. To explicitly associate a build with a merge commit, we run a separate action on push:

name: Publish Storybook

on:
    push:
        branches:
            - main

jobs:
    storybook:
        name: Storybook

        runs-on: ubuntu-latest

        steps:
            - name: Checkout
              uses: actions/checkout@v2
              with:
                  fetch-depth: 0

            - uses: ./.github/actions/load-node-modules

            - name: Create snapshots
              run: yarn chromatic --only-changed --auto-accept-changes
              env:
                  CHROMATIC_PROJECT_TOKEN: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
                  CHROMATIC_SHA: ${{ github.event.after }}
                  CHROMATIC_BRANCH: main

This time we’re specifying the --auto-accept-changes flag to automatically accept the changes, as they have already been reviewed in the context of the PR.

We’re enabling Chromatic’s TurboSnap with the --only-changed flag. TurboSnap uses Webpack’s dependency graph to determine which stories have changed, thus minimizing the amount of snapshots needed per PR. That is especially desirable in the context of a monorepo like ours, since many PRs don’t touch the UI and don’t need to trigger any snapshots. TurboSnap errors on the side of caution though, and if there are changes to package.json, all stories will be considered changed. Since our dependency updates are automated, we use Chromatic’s skip option to mark the visual tests as passed without actually creating any snapshots. It’s possible that updating a dependency will cause UI changes that will go undetected by Chromatic, but right now we’re preferring to conserve the snapshot budget. Note that because we use vanilla-extract for styling, the dependency graph can trace CSS changes to specific stories.

Limitations and pitfalls

As with all tools, there are some non-obvious usages that leave us scratching our heads. These are the ones we repeatedly encounter:

• Snapshotted stories need to be written deterministically to avoid false positives. This means ensuring the absence of randomness and the stability of things like element order and dates: a story that uses Date.now() or shuffles the images in a carousel will always require approval (if snapshotted). Stories are easy enough to fix, but sometimes the non-determinism comes from deep within a component. To help with those, we can tell whether we’re running under Chromatic by using the isChromatic() function.

• Chromatic doesn’t capture animations. Instead, videos and CSS/SVG animations are paused automatically and reset to their initial state. JavaScript animations must be disabled explicitly (isChromatic() is useful here as well). Alternatively, Chromatic can be configured with a delay to allow animations to complete before a snapshot is taken. This doesn’t always solve the problem though. If you’re creating a looping animation (so adding a delay isn’t useful) with a library like framer-motion, which doesn’t expose a way to globally disable animations, then you may need to instruct Chromatic to ignore a DOM element.

• Finally, if using TurboSnap, it’s important to be aware of its limitations. We already mentioned that changes to package.json trigger full snapshots. Another situation that can lead to more snapshots being taken than expected is when stories (or intermediary files) import components through an index file. If any (transitive) import in that index file was changed, then all importers of the index file will be considered changed as well.

Conclusion

Visual regression testing is essential to confidently make changes to a user interface. Front-end development is sufficiently complex that most changes can only be noticed by comparing the rendered interface in a specific viewport and browser. Chromatic makes this very easy by integrating with Storybook, a nearly ubiquitous tool in the JavaScript ecosystem, and layering on top a great review workflow that lets developers comment on and approve or reject changes to an application’s UI.