Automagically Retry Your Flaky E2E Tests with Cypress

End-to-end automated tests are notorious for being flaky. If you've spent some time with automated testing you've likely seen E2E tests that that are just not predictable or consistent. When you go to troubleshoot the failing test, suddenly it's passing. The next morning it has failed again.

If you have a test with results that are not predictable, failures that are impossible to reproduce manually, and root cause analysis that is difficult and time-consuming, then you've got yourself a flaky test.

When these automated tests are blocking your CI pipeline and the failures cannot be reproduced manually, you might consider automatically retrying the test. Retry logic will allow the test to stay in the pipeline and give it a chance to pass.

In the past I've seen custom code written to deal with this type of test. The code will loop some number of times until the test passes or the retry count is met.

Automatically retry a flaky test

Cypress provides a pretty simple way to retry a failing test, without needing to write any code. For example:

it(
    'my flaky test',
    {
      retries: {
        runMode: 2,
        openMode: 0,
      },
    },
    () => {
      cy.visit('https://locahost:3000/')
      // ...and more steps...
    }
  )

See how easy that is?

When using the "cypress run" command, the "runMode" count indicates how many times to retry the test until it passes. The "openMode" count applies when running through the Test Runner with "cypress open".

In "cypress run" mode, this test will be tried a total of 3 times to give it a chance to pass.

Judiciously applying this configuration to your flaky tests can buy you some time while you work to identify the root cause of the flakiness.

Automatically retrying all failing tests

In the previous example we applied the retry to a single test. If you want a more global approach to retries, you can set that up in your "cypress.json" file:

"retries": {
    "runMode": 2,
    "openMode": 0
  }

With this configuration change, ALL failing tests ran using "cypress run" will be attempted a maximum of 3 times.

Why to NOT automatically retry your failing tests

Some tests are flaky, but others are just consistently failing. They may fail because of a known bug, a new bug, or an issue with the automation code, but they all have one thing in common. They ALWAYS fail. Retrying these tests will not help you, and will only result in a longer running test suite.

If a test is consistently failing, there is not point in retrying it.

Wrap-up

As you can see, Cypress provides an easy-to-use mechanism for retrying failed tests. Just remember there is a difference between a FLAKY test and a FAILED test, and use the Cypress retry mechanism appropriately for your situation.

As always, to learn more please check the Cypress doco for more info.

Feel free to subscribe to my blog site for more test automation content. Thanks!