The Most Important Skill When Writing Unit Tests Is Making Sure They Fail

Yes, seriously…

Around one year ago, I came across this LinkedIn post by Craig Livings who posted this controversial statement:

Any test that has never failed adds no value #tdd #agile

What it means is any test that has never failed has no value, and therefore, can be deleted. In order for any test to add value, it must have failed at least once.

Let’s see why failing our tests is such an important skill…

Why must our tests fail?

Let us say we have written some code and some supporting unit tests. We’ve done good right? The tests have never failed, but we now have unit tests that support our code! Or at least we think we have…

Let’s look at some examples. One of a test that has not failed and one that has.

The test that has never failed

This example is a common scenario where we are just focused on writing a test that passes. We have no intention to make it fail.

We will use the following JavaScript function:

function sum(a, b) {
  return a + b;
}

module.exports = sum;

We have written the following Jest test to verify our sum function adds two numbers together:

const sum = require('./sum');

test('two plus two is four', () => {
  expect(2 + 2).toBe(4);
});

We run the test and it passes ✅

That’s it. We’re happy the test has passed so we move on.

The test that has failed

In this example, we use exactly the same code. However, this time we make sure the test fails to prove its value.

Using the same function:

function sum(a, b) {
  return a + b;
}

module.exports = sum;

And using the same test:

const sum = require('./sum');

test('two plus two is four', () => {
  expect(2 + 2).toBe(4);
});

We run the test and it passes ✅

But this time we want to make sure the test has failed. So we decide to modify the code:

function sum(a, b) {
  return 0;
}

module.exports = sum;

We re-run the test and we get another pass ✅

But wait… this should have failed. We removed the code that adds the numbers together.

We can see the reason the test still passed is that the test itself is incorrect. The test was not calling sum.

So we update the test:

const sum = require('./sum');

test('two plus two is four', () => {
  expect(sum(2, 2)).toBe(4);
});

We re-run the test and it fails ❌

Ok, great! Now let’s put our code back:

function sum(a, b) {
  return a + b;
}

module.exports = sum;

And we re-run the test and it passes ✅

Excellent, now our test has value because we have ensured the test is validating the correct thing.

The example is a simple one. However, when testing more complex code, it is difficult to spot these mistakes. This is why it’s so important we make sure our tests fail before we make them pass — so we have confidence in what we are validating.

The takeaway

When writing tests, we must know they have failed, so we know they add value.

Do you agree?

Comments

[Nitzan Hen]

A good point! I can personally point to a few tests that I neglected to check, and eventually turned out to be implemented incorrectly 😬

There's also something called Mutation Testing, which, as far as I know, attempts to automate this exact "testing of tests" you're speaking of - it inserts small changes in the source code, and makes sure your tests fail as a result. I've yet to research it properly but it's quite interesting. For JavaScript, there's a mutation testing tool called Stryker Mutator which seems rather popular.

By the way, It's a matter of nuance, but I think I'd phrase the main statement more as a need to "make sure tests fail when they're supposed to", which is exactly their expected behavior (just like you expect a button to do something when you click it!). The statement "any test that has never failed has no value, and therefore, can be deleted" is a bit radical in that sense, as a test could often be implemented properly, and therefore would fail when it needs to - in other words, it has some value.

Cheers!

[chrisnorthfield]

Thanks @nitzanhen, I'll have to look into mutation testing as I've not come across it before!

As for the statement. Yes it is radical which is why I find it intriguing!

Even though it could be implemented correctly and could fail, if it never has then what value has it added? The only way to prove the value is by it failing.

[Nitzan Hen]

I get what you mean. Intriguing indeed!