DIY: Enhance Code Reviews with Visual Testing

...A Practical Guide Using Playwright and GitHub Actions

🈚️

Introduction

Visual testing is a crucial aspect of UI development that often goes underutilized. In this post, we'll explore how to set up an efficient visual testing workflow using Playwright and GitHub Actions. This approach is particularly beneficial for small to medium-sized teams looking to enhance their code review process and confidence without adding significant overhead or extra costs.

Why Visual Tests Matter

1. Uncover Hidden Issues:
   Visual tests can reveal problems you weren't actively looking for (while also reducing the need for complex assertion code).
   

2. Ensure UI Integrity:
   They're the most effective way to verify the visual aspects of your UI.
   

3. Improve Collaboration:
   Visual tests serve as a great tool for communication between technical and non-technical stakeholders.

The Core Concept: Non-Assertive Visual Testing

Before we dive into the technical details, it's crucial to understand the core concept of our approach:

We're "piggybacking" Playwright tests to collect screenshots, but instead of automated visual assertions, we use the screenshots as an optional tool during the code review process.

Here's why this approach is powerful:

1. Stability: By not making automated assertions based on screenshots, we avoid introducing potential flakiness into our test suite. Our tests remain fast and reliable.

2. Flexibility: During code review, we can choose which screenshot comparisons are relevant to the changes being reviewed. We're not bound by rigid assertions.

3. Context-Aware Reviews: Screenshots provide visual context that might not be apparent from code changes alone, enhancing the effectiveness of code reviews.

This approach strikes a balance between the benefits of visual testing and the need for a smooth, efficient development process. It keeps our CI pipeline stable while still providing valuable visual information during code reviews.

The Solution: Playwright + GitHub Actions

We'll set up a workflow that integrates seamlessly with your existing development process:

1. E2E tests run on every PR using Playwright.
2. Tests capture screenshots only when the PR is ready for review. The screenshots are taken, not asserted on, and don't fail the tests.
3. Screenshots that are now different are compared using ImageMagick to make sure there's an actual visual diff. Otherwise, they are reverted to avoid clutter.
4. GitHub Actions commit these screenshots to the PR.
5. Code reviewers can see image diffs directly in GitHub's file comparison view, using them as an optional aid in their review.
6. Merged PRs set the new baseline for future comparisons.

Bonus Skills You'll Learn:

How to: optimize CI pipelines by creating custom Docker images with preinstalled programs, manipulate PRs through GitHub Actions, and write powerful scripts for CI environments. You'll also gain experience with ImageMagick's image comparison capabilities, enabling programmatic image analysis. These skills will enhance your ability to create efficient, automated workflows and improve your overall DevOps practices.

Let's dive into the implementation!

Step 1: Setting Up Playwright Tests

First, let's configure our Playwright tests to capture screenshots. We'll create a utility function that takes screenshots only for non-draft PRs.

Create a file named test-utils.ts in your test directory:

const skipScreenshots = process.env.DRAFT === 'true';

export const screenshot = async (target: Page | Locator, name: string) => {
  if (skipScreenshots) return;

  await target.screenshot({
    path: `e2e/screenshots/${name}.png`,
    animations: 'disabled',
  });
};

Now, you can use this screenshot function in your tests to capture screenshots without making assertions:

import { test } from '@playwright/test';
import { screenshot } from './test-utils';

test('homepage visual test', async ({ page }) => {
  await page.goto('https://your-app-url.com');
  await screenshot(page, 'homepage');
});

Step 2: Creating a Custom Docker Image

To avoid reinstalling Playwright and ImageMagick on every CI run, we'll create a custom Docker image. This helps speed up the GitHub Actions workflow by ensuring all necessary tools are pre-installed.

Create a file named Dockerfile in your project root:

# Use the official Playwright Docker image as a base
FROM mcr.microsoft.com/playwright:v1.43.0-jammy

# Install ImageMagick
RUN apt-get update && apt-get install -y imagemagick

# Clean up
RUN apt-get clean && rm -rf /var/lib/apt/lists/*

# Display installed versions
RUN playwright --version && identify -version

Now, build and push this Docker image to a registry. If you're using GitHub Container Registry, you can use these commands:

# Build the image
docker build -t ghcr.io/your-username/playwright-imagemagick:latest .

# Log in to GitHub Container Registry
echo $GITHUB_TOKEN | docker login ghcr.io -u your-username --password-stdin

# Push the image
docker push ghcr.io/your-username/playwright-imagemagick:latest

Make sure to replace your-username with your actual GitHub username and set the GITHUB_TOKEN environment variable to a personal access token with the necessary permissions.

Step 3: Configuring GitHub Actions

Next, we'll update our GitHub Actions workflow to use our custom Docker image. Create/update a file named .github/workflows/e2e-tests.yml:

name: E2E Tests
on:
  pull_request:
    types: [opened, reopened, synchronize, ready_for_review]

jobs:
  e2e:
    name: 'E2E Tests'
    timeout-minutes: 60
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/your-username/playwright-imagemagick:latest
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Run Playwright tests
        run: npm run test:e2e
        env:
          # Skip screenshots for draft PRs or non-main branch PRs
          DRAFT: ${{ github.event.pull_request.draft || github.base_ref != 'main' }}

      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: screenshots
          path: e2e/screenshots
          retention-days: 3

  commit-screenshots:
    name: 'Commit Screenshots'
    needs: e2e
    runs-on: ubuntu-latest
    if: github.event.pull_request.draft == false && github.base_ref == 'main'
    permissions:
      contents: write
    container:
      image: ghcr.io/your-username/playwright-imagemagick:latest
      credentials:
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - uses: actions/download-artifact@master
        with:
          name: screenshots
          path: e2e/screenshots

      - name: Compare Modified Screenshots
        run: |
          chmod +x $GITHUB_WORKSPACE/scripts/compare_screenshots.sh
          $GITHUB_WORKSPACE/scripts/compare_screenshots.sh

      - name: Commit changes
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: '[CI] Update Screenshots'
          commit_user_name: 'GitHub Actions'
          commit_user_email: '[e-mail]'

      - uses: geekyeggo/delete-artifact@v5
        with:
          name: screenshots

Step 4: Image Comparison Script

Create a file named scripts/compare_screenshots.sh:

#!/bin/bash
set -e

compare_images() {
  local img1="$1"
  local img2="$2"
  local diff_img="$3"

  local diff_pixels
  diff_pixels=$(compare -metric AE "$img1" "$img2" "$diff_img" 2>&1 >/dev/null)

  echo "diff pixels for $img1, $img2: $diff_pixels"

  if [ "$diff_pixels" -lt 11 ]; then
    return 0
  else
    return 1
  fi
}

SCREENSHOT_DIR="${GITHUB_WORKSPACE}/e2e/screenshots"
modified_files=$(git diff --name-only | grep '.png$' || true)

if [[ -z "$modified_files" ]]; then
  echo "No modified PNG files found in $SCREENSHOT_DIR."
  exit 0
fi

for screenshot in $modified_files; do
  current_screenshot="${GITHUB_WORKSPACE}/${screenshot}"
  git show HEAD~1:"$screenshot" > original_screenshot.png
  baseline_screenshot="original_screenshot.png"

  if [[ ! -s "$baseline_screenshot" ]]; then
    echo "No baseline image for $screenshot. Assuming this is a new screenshot."
    rm -f "$baseline_screenshot"
    continue
  fi

  diff_file="${current_screenshot}.diff.png"
  set +e
  compare_images "$baseline_screenshot" "$current_screenshot" "$diff_file"
  comparison_result=$?
  set -e

  if [ $comparison_result -eq 0 ]; then
    echo "No visual difference for $screenshot. Reverting changes."
    git restore "$screenshot"
  else
    echo "Visual difference detected for $screenshot. Keeping changes."
  fi

  rm -f "$diff_file" "$baseline_screenshot"
done

This script compares the new screenshots with the previous versions and reverts changes if the difference is minimal (less than 11 pixels).

Step 5: Setting Up Secrets

To allow the GitHub Action to commit changes, you need to set up a personal access token:

1. Go to your GitHub account settings.
2. Navigate to "Developer settings" > "Personal access tokens".
3. Generate a new token with repo scope.
4. In your repository, go to "Settings" > "Secrets and variables" > "Actions".
5. Add a new repository secret named GITHUB_TOKEN with the value of your personal access token.

Step 6: Bonus - Enhancing Image Comparison in GitHub

To make comparing screenshots even easier, I’m working on an open source browser extension that improves GitHub’s built-in image diff tool. This extension will highlight visual differences more clearly, making it easier for reviewers to assess changes.

Stay tuned for the release!

Advantages of This Approach

1. Integrated Workflow: Utilizes existing tools and processes.
2. Enhanced Code Reviews: Visual diffs are part of the PR review process, providing additional context without enforcing rigid rules.
3. No Additional Costs: Leverages GitHub Actions and your existing CI pipeline.
4. Flexible: Can be easily adapted to different project sizes and needs.

Limitations and Considerations

1. Repository Size: Frequent screenshots can increase repository size over time.
2. Review Time: Large numbers of screenshots may slow down the review process.
3. Limited Comparison Tools: GitHub's built-in image diff tool is basic.
4. Manual Baseline Maintenance: It's up to you, the reviewer, to go through the diffs and check them with your own eyes - with no AI or smart algorithms to save you time on recurring diffs.
5. Pulling Changes: After the CI commits screenshots, developers need to pull the changes to sync their local repository.

Conclusion

This setup provides a practical approach to incorporating visual testing into your development workflow, without the overhead of maintaining visual test assertions. By using screenshots as an optional review aid rather than a pass/fail criterion, we gain the benefits of visual testing while maintaining a smooth, efficient development process.

Remember, the key is to use these visual diffs as a tool to enhance your code reviews, not as a replacement for thorough code analysis. This approach allows you to catch visual regressions early, while still relying on human judgment to determine the significance of any visual changes.

As your team and project grow, you might consider more advanced solutions like Applitools, which offer powerful image comparison algorithms and cross-browser testing capabilities.

Happy testing, and please feel free to ask any question!

I encourage you to implement this in your workflow and share your experiences in the comments.