Parallel tests on circleci

When you have a large codebase, the tests suite that in the beginning used to take less than 5 minutes to run, starts to take more than 30 minutes to run. That is the case of our main project, we have a automated build that runs brakeman, rubocop, bundler audit and tests configured on circleci to run for every PR opened on github and every commit pushed to master too, and the builds were tooking more than 40 minutes to run.

This always bothered me, but in the last weeks I began to be even more bothered with that. The main reason for that change is that we change some practices and culture here in SumOne to make the developers team care more with the final delivery and to create a more agile team, a faster build and delivery process, so we are able to deliver more value to ours clients. So, ours developers now care about and are involved in every step of our process, since the discovery until delivery, and have to wait more than 40 minutes to be able to have yours pull requests approved to be merged were not a agile and faster process in the middle of all this.

So, I started to think about how to solve this. Some months ago I tried to use a gem to run tests in parallel locally, but it did not work fine. So, I decided to have just one goal, how to make the build faster and the tests run in parallel just on circleci, and give up for faster local tests for now.

First of all, you need to decide how many parallel jobs you want, and unfortunately circleci does not allow parallelism on free account. So, for example, if you decide to run 2 jobs in parallel, you will need to add a paid container that costs $50 dollars. After that, just configure .circleci/config.yml with parallelism: 2 .

The next step is to run different files for rspec for each container. Looking for the documentation and support conversations of circleci, I ended up with those lines in my .circleci/config.yml project file:

- run:
  name: Tests
  command: |
    TESTFILES=$(circleci tests glob "spec/\*\*/\*.rb" | circleci tests split --split-by=timings)
    bundle exec rspec -- ${TESTFILES}

The first line first get all the files inside spec folder using circleci tests glob command and then split it using the circleci tests split command. There is some options to decide how to split the tests, you can find it here in documentation. I decided to use the --split-by=timings to let circleci take care and find the best way of how to run the tests faster. The output of the first line is a list of files, and it is a different list for each circleci container that is running your build.

The second line is easier to understand for rails developers that are used to rspec test library. We just call rspec command passing to it the list of files, so that way we have each container running different tests.

Easier right? Almost! After I've done that, I found the first problem. We use factory_bot in our project, and circleci tests glob "spec/**/*.rb are getting the files inside the spec/factories folder, and when it is passed to bundle exec rspec as argument, rspec tried to run the factories file and we got a Factory FACTORY_NAME already registered error and nothing works.

So, another look at circleci documentation and I found this section where they show the options they allow you to pass to circleci tests glob as params to filter the files. So, I changed again the .circleci/config.yml file and the result was:

- run:
  name: Tests
  command: |
    TESTFILES=$(circleci tests glob "spec/{controllers,features,helpers,listeners,mailers,models,services,workers}/\*\*/\*.rb" | circleci tests split --split-by=timings)
    bundle exec rspec -- ${TESTFILES}

So, now the circleci only filters by real test files and pass it as params to rspec command. After commit and push it to github, circleci ran it again and the tests suites that were taken more than 40 minutes starts to run in less than 10 minutes.

One last thing, I decided to do that for rubocop task too, so I added those lines to the config file:

- run:
  name: Rubocop
  command: |
    PROJECTFILES=$(circleci tests glob "app/\*\*/\*.rb" | circleci tests split --split-by=timings)
    bundle exec rubocop -- ${PROJECTFILES}

Rubocop task time was not a big issue, but since I did it for rspec, there was no reason to not do this with rubocop, and the result was that a task that was taken almost 2 minutes is now takin less than 20 seconds to run.

So, that was the easy way to improve our process and let the build faster, the next steps are to really improve our tests, try to make every single test faster to run.

---