Doing TDD is an integral part of my day to day workflow. Tests help me to break down complex problems into smaller chunks which I can process more easy. This helps me to develop parts of the application in isolation and focus on the core business logic without the fear of breaking existing functionality. Getting fast feedback to my changes empowers me to move fast and build more robust systems. Having lots of small tests which check one behavior of a unit under test makes it easy to see what the application is capable of. Quite often those small tests cause a maintenance overhead as additional requirements make those tests break, even though functionality was only added and existing behavior was kept untouched. This leads to the issue that tests need to be altered even though their behavior didn't change. Another issue often arises when writing tests against external systems like databases.
If an order by clause is not specified, then the ordering of the rows of Q is implementation-dependent. (https://dba.stackexchange.com/a/6053)
Getting records back in a different order each test run is a common issue and might result in green suite locally but failing tests on CI. After some research I realized that the way my test assertions were written, might be the root cause of my brittle tests. In this post I'll share some of my finding on my journey to write better test assertions.
TLDR
Watch the talk from the Vienna JS Meetup in double speed.
What is a test assertion
An assertion is a boolean expression at a specific point in a program which will be true unless there is a bug in the program Source. A very basic implementation of an assertion might look similar to the following:
const assert = (value, message = 'assertion failed') => {
if (!value) { throw new Error(message); }
}
assert(1 === 1, '1 should be equal to 1');
assert(1 === 2, '1 should be equal to 1'); // Throws exception
Whenever a falsy value is passed to the assert function an exception is thrown with an optional message. When an unhandled exception is thrown inside a test-case, it is automatically marked as failed. The above test assertion is very low level and not very expressive. Assertion libraries solve this problem by providing a variety of different high level assertions which make the test easier to read. Some common assertion libraries include:
- chai.js (https://www.chaijs.com/)
- node.js assert module (https://nodejs.org/api/assert.html)
- should.js (http://shouldjs.github.io/)
- hamjest (https://github.com/rluba/hamjest)
The scenario
To make the issue with "hard to maintain" tests easier to understand. I created an artificial application with different user stories. The application is an employee management system for a local supermarket. The owner wants to open the supermarket on Sundays and due to legal constraints not all employees are allowed to work on Sundays. To see who is allowed to work she asked to generate a special report of her employees. Out of simplicity the implementation focuses on the business logic in JS only. In a real world application one might query the database directly.
First User-Story
As shop owner I want to view a list of all employees, which are older than 18 years, so that I know who is allowed to work on Sundays.
After reading this requirement the following test case is generated.
import { assertThat, equalTo } from 'hamjest';
const employees = [
{ name: 'Max', age: 17 },
{ name: 'Sepp', age: 18 },
{ name: 'Nina', age: 15 },
{ name: 'Mike', age: 51 }
];
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[1], employees[3]]));
});
After running the tests the following test fails:
❌ returns employees which are older than 18
To make this test green the following function is implemented:
const listEmployees = (employees) => employees
.filter((employee) => employee.age >= 18);
After running the tests again the test shows green.
✔️ returns employees which are older than 18
Second User-Story
As shop owner I want the list of employees to be sorted by their name, so I can find employees easier.
Without looking to much on the existing test the next test case is added:
import { assertThat, equalTo } from 'hamjest';
const employees = [
{ name: 'Max', age: 17 },
{ name: 'Sepp', age: 18 },
{ name: 'Nina', age: 15 },
{ name: 'Mike', age: 51 }
];
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[1], employees[3]]));
});
// New test Case
it('returns employees ordered by their name', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[3], employees[1]]));
});
✔️ returns employees which are older than 18
❌ returns employees ordered by their name
After watching the new test fail, the following is implemented:
const listEmployees = (employees) => employees
.filter((employee) => employee.age >= 18)
.sort((a, b) => a.name.localeCompare(b.name));
❌ returns employees which are older than 18
✔️ returns employees ordered by their name
The sorting functionality was implemented successfully but now the first already working test is failing. After comparing the test assertions it is obvious why the test fails. The test might be changed as follows:
// before
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[1], employees[3]]));
});
// afterwards
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, containsInAnyOrder(employees[1], employees[3]));
});
The containsInAnyOrder matcher fixes the previous issue by ignoring the sorting of the result. It verifies that the two elements need to be present independent their order. This change results in a green test suite.
️✔️ returns employees which are older than 18
✔️ returns employees ordered by their name
Third User-Story
As shop owner I want the list of employees to be capitalized, so I can read it better.
Starting from the test file again a new test is added:
import { assertThat, equalTo, containsInAnyOrder } from 'hamjest';
const employees = [
{ name: 'Max', age: 17 },
{ name: 'Sepp', age: 18 },
{ name: 'Nina', age: 15 },
{ name: 'Mike', age: 51 }
];
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, containsInAnyOrder(employees[1], employees[3]));
});
it('returns employees ordered by their name', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[3], employees[1]]));
});
// New test case
it('returns employees whose names are capitalized', () => {
const result = listEmployees(employees);
assertThat(result[0].name, equalTo('MIKE'));
assertThat(result[1].name, equalTo('SEPP'));
});
✔️ returns employees which are older than 18
✔️ returns employees ordered by their name
❌ returns employees whose names are capitalized
One possible implementation to satisfy the failing this looks like this:
const listEmployees = (employees) => employees
.filter((employee) => employee.age >= 18)
.sort((a, b) => a.name.localeCompare(b.name))
.map((employee) => ({ ...employee, name: employee.name.toUpperCase() }));
After running the tests we see that the new behavior was added successfully but we broke all the other tests.
❌️ returns employees which are older than 18
❌️ returns employees ordered by their name
✔️ returns employees whose names are capitalized
The issue with the other tests is that hamjest can't compare the objects anymore because the capitalized names differ from the original ones. In this trivial example changing 2 tests might not be the biggest issue. In a more complex example figuring out if the the change broke the original behavior might take more time. In this example the test might be changed to:
// original test
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[1], employees[3]]));
});
// first iteration
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
result.forEach((employee) => {
assertThat(employee.age >= 18, equalTo(true));
});
});
// final iteration
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, everyItem(hasProperty('age', greaterThanOrEqualTo(18))));
});
By changing the assertion to the following we introduced one major issue to this test. The following implementation results in a green test.
const listEmployees = (employees) => []
So this assertion is now 'underspecified', which means that an invalid/broken implementation results in a green testsuite. By changing the assertion to the following, one can prevent this:
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, allOf(
hasProperty('length', greaterThanOrEqualTo(1)),
everyItem(hasProperty('age', greaterThanOrEqualTo(18))),
);
});
✔️ returns employees which are older than 18
❌️ returns employees ordered by their name
✔️ returns employees whose names are capitalized
The other test might be changed to:
// original implementation
it('returns employees ordered by their name', () => {
const result = listEmployees(employees);
assertThat(result, equalTo([employees[3], employees[1]]));
});
// final iteration
it('returns employees ordered by name', () => {
const result = listEmployees(employees);
assertThat(result, orderedBy((a, b) => a.name < b.name));
});
After those changes, all 3 tests are green. As the empty result issue is already checked by the previous test we don't test this behaviour in the other tests.
✔️ returns employees which are older than 18
✔️ returns employees ordered by their name
✔️ returns employees whose names are capitalized
Fourth User-Story
As shop owner I want the employees to be sorted by their names descending instead of ascending.
As there is already a test-case which verifies the order, we decide to change this test to match the new requirements.
import {
assertThat,
greaterThanOrEqualTo,
everyItem,
orderedBy,
hasProperty,
} from 'hamjest';
const employees = [
{ name: 'Max', age: 17 },
{ name: 'Sepp', age: 18 },
{ name: 'Nina', age: 15 },
{ name: 'Mike', age: 51 },
];
it('returns employees which are older than 18', () => {
const result = listEmployees(employees);
assertThat(result, everyItem(hasProperty('age', greaterThanOrEqualTo(18))));
});
// changed assertion
it('returns employees ordered by name descendent', () => {
const result = listEmployees(employees);
assertThat(result, orderedBy((a, b) => a.name > b.name));
});
it('returns employees whose names are capitalized', () => {
const result = listEmployees(employees);
assertThat(result[0].name, equalTo('MIKE'));
assertThat(result[1].name, equalTo('SEPP'));
});
✔️ returns employees which are older than 18
️️❌ returns employees ordered by their name descendent
️️️✔️ returns employees whose names are capitalized
To make our test green again the following code is implemented:
const listEmployees = (employees) => employees
.filter((employee) => employee.age >= 18)
.sort((a, b) => b.name.localeCompare(a.name))
.map((employee) => ({ ...employee, name: employee.name.toUpperCase() }));
The third test reports a failure now.
✔️ returns employees which are older than 18
✔️ returns employees ordered by their name descendent
️️️️️❌ returns employees whose names are capitalized
// original implementation
it('returns employees whose names are capitalized', () => {
const result = listEmployees(employees);
assertThat(result[0].name, equalTo('MIKE'));
assertThat(result[1].name, equalTo('SEPP'));
});
// first iteration
it('returns employees whose names are capitalized', () => {
const result = listEmployees(employees);
assertThat(result, everyItem(hasProperty('name', matchesPattern(/[A-Z]*/))));
});
// second iteration
const inUpperCase = () => matchesPattern(/[A-Z]*/);
it('returns employees whose names are capitalized', () => {
const result = listEmployees(employees);
assertThat(result, everyItem(hasProperty('name', inUpperCase())));
});
We run the tests and see that all tests are green.
✔️ returns employees which are older than 18
✔️ returns employees ordered by their name descendent
️️️️️✔️ returns employees whose names are capitalized
Conclusion
This blog post showed that additional requirements might cause existing tests to fail even though their behavior didn't change. By expressing the exact desired result in an assertion makes the overall test-suite less brittle and easier to change. Having tests which don't depend on implementation details or previous tests makes it easier to add and remove functionality. For example a new feature request where employees should be returned randomized is not causing other tests to break. I've been using hamjest for the last couple of years and I can recommend to test it out.
Edit
I refactored the examples with jest and chai and pushed them to https://github.com/webpapaya/better-test-assertions. Both jest-expect and chai has issues when matching elements in an array. The API design from hamjest is easy to extend and makes it possible to write extremly complex matchers.
Edit2
The drawback section was replaced by a better matcher as it doesn't introduce a new test.
Top comments (1)
Great post and great video! I really like it! Thanks a lot for that!