Export test run order when using ember-exam

One of the features of ember-exam is that it will allow you to run your tests in random order. Which is great to uncover code that is leaking state in your application or tests.

Leaking state is tricky to uncover because it will stay hidden for most of the time, until something changes. In my case the change is usually caused by different order of tests being run. And therefore the state of one component / test starts leaking into other test run.

My primitive strategy is to take the test that started $suddenlyFailing and look at all tests that ran before. One of them has to be the offender. This creates our initial $offendingList.

The nice way to find the order of tests is to print to the console when one is starting / finishing. To do that, just add to your tests/test-helper.js:

import Application from '../app';
import config from '../config/environment';
import { setApplication } from '@ember/test-helpers';
import QUnit from 'qunit';
import { start } from 'ember-qunit';

setApplication(Application.create(config.APP));

// START custom code
QUnit.on( 'testStart', function( data ) {
  console.log( 'Start: ' + data.fullName );
} );
QUnit.on( 'testEnd', function( data ) {
  console.log( 'End: ' + data.fullName + ' ---- ' + data.status );
} );
// END custom code

start();

As a bonus exercise you can run ember exam --random=[SEED] several times for different values of SEED and remove from the $offendingList all the tests:

• that ran before $suddenlyFailing in the case when our test did not fail
• that ran after $suddenlyFailing in the case when our test did fail