Recently I've been asked to work on a solution of efficiently running Cypress component tests on pull requests without taking a lot of time. At first, my standing solution was to just evenly spread out the files against a number of parallel jobs on GitHub Actions workflows, but there is a big discrepancy between the slowest job and the average job times. Thus, we've been wondering if there is a smarter way of evening out the runtimes.
With that, I created a new plugin of cypress-load-balancer, which allows us to solve that problem. This plugin saves the durations of the tests it runs and calculates an average, which then then can be passed into a script; that script uses an algorithm to perform load balancing for a number of job runners.
What is a load balancer?
Wikipedia's summary is as such:
In computing, load balancing is the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient. Load balancing can optimize response time and avoid unevenly overloading some compute nodes while other compute nodes are left idle.
The general approach of using a load balancer for tests
This is the basic idea of steps that need to occur to utilize results from load balancing properly. A persistent load balancing map file known as spec-map.json is saved on the host machine. The load balancer will reference that file and perform calculations to assign tests across a given number of runners. After all parallel test jobs complete, they will create a key-value list of test file names to their execution time; these results can then be merged back to the main spec map file, recalculate a new average duration per each test file, and then overwrite the original file on the host machine. Then the spec map can be consumed on the next test runs, and run through this process all over and over again.
For this tool, here are the general steps:
-
Install and configure the plugin in the Cypress config. When Cypress runs, it will be able to locally save the results of the spec executions per each runner, depending on
e2eorcomponenttests. - Initialize the load balancer main map file in a persisted location that can be easily restored from cache. This means the main file needs to be in a place outside of the parallelized jobs to can be referenced by the parallelized jobs in order to save new results.
- Execute the load balancer against a number of runners. The output is able to be used for all parallelized jobs to instruct them which specs to execute.
- Execute each parallelized job that starts the Cypress testrunner with the list of spec files to run across each runner.
- When the parallelized jobs complete, collect and save the output of the load balancing files from each job in a temporary location.
- After all parallelized test jobs complete, merge their load balancing map results back to the persisted map file and cached for later usage. This is where the persisted file on the host machine gets overwritten with new results to better perform on the next runs. (In a GitHub Actions run, this means on pull request merge, the load balancing files from the base branch and the head branch need to be merged, then cached down to the base branch.)
So, for Docker Compose, a persistent volume needs to exist for the host spec-map.json to be saved. It can then run the load balancing script, and execute a number of parallelized containers to run those separated Cypress tests. When each test job completes, the duration of each test can be merged back to the original file and re-calculate a new average.
For GitHub Actions, it's a bit more complex. More on that later.
How does it work for Cypress automated tests?
Installation
The current installation guide as of February 2025 is as such:
Install the package to your project:
npm install --save-dev cypress-load-balancer yarn add -D cypress-load-balancerAdd the following to your
.gitignoreand other ignore files:.cypress_load_balancerIn your Cypress configuration file, add the plugin separately to your
e2econfiguration and alsocomponent
configuration, if you have one.
This will register load balancing for separate testing typesimport { addCypressLoadBalancerPlugin } from "cypress-load-balancer"; defineConfig({ e2e: { setupNodeEvents(on, config) { addCypressLoadBalancerPlugin(on); } }, component: { setupNodeEvents(on, config) { addCypressLoadBalancerPlugin(on); } } });
Usage
- Cypress tests are run for e2e or component testing types.
- When the run completes, the durations and averages of all executed tests are added to
spec-map.json. - The
spec-map.jsoncan now be used by the included executable,cypress-load-balancer, to perform load balancing against the current Cypress configuration and tests that were executed. The tests are sorted from slowest to fastest and then assigned out per runner to get them as precise as possible to each other in terms of execution time. For example, with 3 runners and e2e tests:npx cypress-load-balancer --runners 3 --testing-type e2e
- The script will output an array of arrays of spec files balanced across 3 runners.
Scripts
There are included scripts with npx cypress-load-balancer:
shell
$: npx cypress-load-balancer --help
cypress-load-balancer
Performs load balancing against a set of runners and Cypress specs
Commands:
cypress-load-balancer Performs load balancing against a set of
runners and Cypress specs [default]
cypress-load-balancer initialize Initializes the load balancing map file and
directory.
cypress-load-balancer merge Merges load balancing map files together
back to an original map.
Options:
--version Show version number [boolean]
-r, --runners The count of executable runners to use
[number] [required]
-t, --testing-type The testing type to use for load balancing
[string] [required] [choices: "e2e", "component"]
-F, --files An array of file paths relative to the current
working directory to use for load balancing.
Overrides finding Cypress specs by configuration
file.
If left empty, it will utilize a Cypress
configuration file to find test files to use for
load balancing.
The Cypress configuration file is implied to
exist at the base of the directory unless set by
"process.env.CYPRESS_CONFIG_FILE"
[array] [default: []]
--format, --fm Transforms the output of the runner jobs into
various formats.
"--transform spec": Converts the output of the
load balancer to be as an array of "--spec
{file}" formats
"--transform string": Spec files per runner are
joined with a comma; example:
"tests/spec.a.ts,tests/spec.b.ts"
"--transform newline": Spec files per runner are
joined with a newline; example:
"tests/spec.a.ts
tests/spec.b.ts"
[choices: "spec", "string", "newline"]
--set-gha-output, --gha Sets the output to the GitHub Actions step output
as "cypressLoadBalancerSpecs" [boolean]
-h, --help Show help [boolean]
Examples:
Load balancing for 6 runners against cypressLoadBalancer -r 6 -t
"component" testing with implied Cypress component
configuration of `./cypress.config.js`
Load balancing for 3 runners against cypressLoadBalancer -r 3 -t e2e -F
"e2e" testing with specified file paths cypress/e2e/foo.cy.js
cypress/e2e/bar.cy.js
cypress/e2e/wee.cy.js
Example on GitHub Actions
I included two workflows in the package that show how this can work for tests executed on pull requests.
Generally, here is what occurs:
Running tests on pull requests
- get-specs:
- A cached load balancing map is attempted to be restored. It tries for the current target branch, then for the source branch, and if none can be found, it initializes a basic map of the files to be run.
- Load balancing is performed based on the user's input of the number of jobs to use. It outputs an array of specs for each runner.
- cypress_run_e2e:
- These are the parallelized jobs that run a subset of the files obtained from the load balancer output.
- When this job completes, it produces a temporary
spec-map.jsonof just those files, and uploads the artifact.
- merge_cypress_load_balancing_maps:
- After all parallel jobs complete, download their artifacts of their temporary
spec-map.jsonfiles, merge them to the branch's map file, and then cache and upload it. This is how it can be saved per branch.
- After all parallel jobs complete, download their artifacts of their temporary
yml
name: Testing load balancing Cypress E2E tests
on:
pull_request:
workflow_dispatch:
inputs:
runners:
type: number
description: Number of runners to use for parallelization
required: false
default: 3
debug:
type: boolean
description: Enables debugging on the job and on the cypress-load-balancer script.
env:
runners: ${{ inputs.runners || 3}}
CYPRESS_LOAD_BALANCER_DEBUG: ${{ inputs.debug || false }}
jobs:
get_specs:
runs-on: ubuntu-22.04
outputs:
e2e_specs: ${{ steps.e2e-cypress-load-balancer.outputs.cypressLoadBalancerSpecs }}
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- run: |
yarn install
yarn build
- name: Get cached load-balancing map
id: cache-restore-load-balancing-map
uses: actions/cache/restore@v4
with:
fail-on-cache-miss: false
path: .cypress_load_balancer/spec-map.json
key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
# Restore keys:
## 1. Same key from previous workflow run
## 2. Key from pull request base branch most recent workflow. Used for the "base" map, if one exists
restore-keys: |
cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-
cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-
cypress-load-balancer-map-${{ github.base_ref }}-
- name: Perform load balancing for E2E tests
id: e2e-cypress-load-balancer
#TODO: this can eventually be replaced with a GitHub action. The executable should be used for Docker and other CI/CD tools
run: npx cypress-load-balancer -r ${{ env.runners }} -t e2e --fm string --gha
#run: echo "specs=$(echo $(npx cypress-load-balancer -r ${{ env.runners }} -t e2e --fm string | tail -1))" >> $GITHUB_OUTPUT
- name: "DEBUG: read restored cached spec-map.json file"
if: ${{ env.CYPRESS_LOAD_BALANCER_DEBUG == 'true' }}
run: cat .cypress_load_balancer/spec-map.json
cypress_run_e2e:
runs-on: ubuntu-22.04
needs: get_specs
strategy:
fail-fast: false
matrix:
spec: ${{ fromJson(needs.get_specs.outputs.e2e_specs) }}
steps:
- name: Generate uuid to use uploading a unique load balancer map artifact
id: generate-uuid
run: echo uuid="$(uuidgen)" >> $GITHUB_OUTPUT
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- name: Cypress run e2e tests
uses: cypress-io/github-action@v6
with:
browser: electron
build: yarn build
spec: ${{ matrix.spec }}
# Fix for https://github.com/cypress-io/github-action/issues/480
config: videosFolder=/tmp/cypress-videos
- name: Upload temp load balancer map
if: always()
uses: actions/upload-artifact@v4
with:
name: ${{steps.generate-uuid.outputs.uuid }}-cypress-load-balancer-map-temp-from-parallel-job
path: .cypress_load_balancer/spec-map.json
merge_cypress_load_balancing_maps:
runs-on: ubuntu-22.04
needs: [get_specs, cypress_run_e2e]
if: ${{ needs.get_specs.result == 'success' }}
steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- run: |
yarn install
yarn build
- name: Get cached load-balancing map
id: cache-restore-load-balancing-map
uses: actions/cache/restore@v4
with:
fail-on-cache-miss: false
path: .cypress_load_balancer/spec-map.json
key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
# Restore keys:
## 1. Same key from previous workflow run
## 2. Key from pull request base branch most recent workflow
restore-keys: |
cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-
cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-
cypress-load-balancer-map-${{ github.base_ref }}-
- name: If no map exists for either the base branch or the current branch, then initialize one
id: initialize-map
run: npx cypress-load-balancer initialize
if: ${{ hashFiles('.cypress_load_balancer/spec-map.json') == '' }}
- name: Download temp maps
uses: actions/download-artifact@v4
with:
pattern: "*-cypress-load-balancer-map-temp-from-parallel-job"
path: ./cypress_load_balancer/temp
merge-multiple: false
- name: Merge files
run: npx cypress-load-balancer merge -G "./cypress_load_balancer/temp/**/spec-map.json"
- name: Save overwritten cached load-balancing map
id: cache-save-load-balancing-map
uses: actions/cache/save@v4
with:
#This saves to the workflow run. To save to the base branch during pull requests, this needs to be uploaded on merge using a separate action
# @see `./save-map-on-to-base-branch-on-pr-merge.yml`
key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
path: .cypress_load_balancer/spec-map.json
# This is to get around the issue of not being able to access cache on the base_ref for a PR.
# We can use this to download it in another workflow run: https://github.com/dawidd6/action-download-artifact
# That way, we can merge the source (head) branch's load balancer map to the target (base) branch.
- name: Upload main load balancer map
if: always()
uses: actions/upload-artifact@v4
with:
name: cypress-load-balancer-map
path: .cypress_load_balancer/spec-map.json
- name: "DEBUG: read merged spec-map.json file"
if: ${{ env.CYPRESS_LOAD_BALANCER_DEBUG == 'true' }}
run: cat .cypress_load_balancer/spec-map.json
Merging back on pull requests
- When the pull request is merged, the newest map uploaded from the source branch's testing workflow is downloaded, merged with the base branch's map, and then cached to the base branch. This allows it to be reused on new pull requests to that branch.
yml
# See https://github.com/brennerm/github-actions-pr-close-showcase/
name: Save load balancing map from head branch to base branch on pull request merge
on:
pull_request:
types: [closed]
jobs:
save:
# this job will only run if the PR has been merged
if: github.event.pull_request.merged == true
runs-on: ubuntu-latest
steps:
- run: |
echo PR #${{ github.event.number }} has been merged
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
- run: |
yarn install
yarn build
- name: Download load-balancing map from head branch using "cross-workflow" tooling
id: download-load-balancing-map-head-branch
uses: dawidd6/action-download-artifact@v8
with:
workflow: cypress-parallel.yml
# Optional, will get head commit SHA
pr: ${{ github.event.pull_request.number }}
name: cypress-load-balancer-map
path: .cypress_load_balancer
- name: Restore cached load-balancing map on base branch
id: cache-restore-load-balancing-map-base-branch
uses: actions/cache/restore@v4
with:
fail-on-cache-miss: false
path: /temp/.cypress_load_balancer/spec-map.json
key: cypress-load-balancer-map-${{ github.base_ref }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: |
cypress-load-balancer-map-${{ github.base_ref }}-
- name: Merge files
run: npx cypress-load-balancer merge -G "./temp/.cypress_load_balancer/spec-map.json"
- name: Save merged load-balancing map
uses: actions/cache/save@v4
with:
path: .cypress_load_balancer/spec-map.json
key: cypress-load-balancer-map-${{ github.base_ref }}-${{ github.run_id }}-${{ github.run_attempt }}
And that's it! This is probably a very niche example, but the general approach should be the same:
- Save a spec map on the host machine
- Perform load balancing against the spec map
- Run parallel test jobs organized by the list of files separated by the load balancer
- Collect their results
- Merge those results back to the host map and recalculate the average
- Repeat!
Top comments (0)