Retrying "flickering" tests with Scala test library (The Retries library)

Writing tests in Scala, either Unit tests or Integration tests, has it's own challenges, flickering tests is one of them.

“flickers”— tests that usually pass, but occasionally fail. The best way to deal with such tests is to fix them so they always pass. Sometimes, however, this is not practical. In such cases, flickers can waste your time by forcing you to investigate test failures that turn out to be flickers. Or worse, like the boy who cried wolf, the flickers may train you an your colleagues to not pay attention to failures such that you don't notice real problems, at least not in a timely manner.
Scala, the retries library docs

Imagine you need an integration test that depends on a chain of microservices processing dat.

1. A service in charge of processing a POST request that later is sentto a broker like Kafka.
2. A service that process this data and enrich this data with other coming from multiple sources (databases, borkers, etc).
3. A service that finally provides the result of this processing in a Rest service.

The integration test needs then to verify that after sending this data to the first service, the data arrives with some assertions from the last service in the chain, a simple GET request is perform on it and the body should provide some specific data. This can easily turn into a flickering test, as described in the documentation, a test that in our best efforts, will perform as expected but sometimes, might need some extra time to be processed.

In my scenario I needed to use the Retries library coming from the Scala test libraries in order to provide some conditions that should be matched before I introduce the assertions in the data that I expect.

 Scala
Class TestExample extends AnyWordSpecLike with Retries{

// TEST FIXTURE CONFIG
  override def withFixture(test: NoArgTest): Outcome = {
    if (isRetryable(test)) {
      val retryDelay: Span = Span.convertDurationToSpan(1, duration.SECONDS)

      //retry strategy
      withRetryOnCancel(retryDelay)(super.withFixture(test))
    }
    else {
      super.withFixture(test)
    }
  }



//TESTS 

"This test should"  taggedAs tagobjects.Retryable in {

 // YOUR TEST HERE

}

The Retries library will support you to handle this scenario by letting you tag some of the tests you consider should have a different treatment. The first step is to extend the test Class with the Retries library, this library will now provide you the capability retrying our tests with some different strategies, the one I am using is call 'withRetryOnCancel'

In the documentation 'withRetryOnCancel' is described as a test that will succeed after retrying only if we have a clear understanding of what are the rules for our test to be cancel, this allow us to take control, like in my case, of the conditions that might not be satisfied in my test in order to properly define the assertions I am expecting.

The tests can be defined then as:

 Scala
"My test should pass" taggedAs Retryable in {

      if (myconditionAreNotValid) {
        cancel("Conditions are not valid")

      } else {
        myAssertions
      }
}  

In my scenario, I needed to verify that after making a GET request to a service, the body of the response should first contain a 200 code, only after I introduce the assertions that I needed for this test, using 'withRetryOnCancel' I was able to cancel any test that was not providing the required code.

There could be multiple scenarios in which flickering tests will turn to be a problem, specially when defining integration tests, in my use case I was able later on to improve the way I retry this tests in order to provide some specific amount of time that this retry mechanism was going to be executed, eventually It cancel its execution when the conditions are not matching for an specific amount of time after running the tests.

I hope my use case could be useful for some other people too.