Fix bugs ⚡ fast with regression tests and auto-bisect

Even if you have amazing test coverage (don’t we all?), chances are you’ll still run into unexpected behavior in your application. You may have missed a case, or you may not have thought it was crucial to test at the time. Whatever the reason, it can be tricky to debug unexpected behavior after the fact. Once you do find and fix the bug, you might create a regression test to prevent it from happening again…

… but did you know you can write the regression test first and make it debug for you?

NOTE: If you haven’t heard of atomic commits or interactive rebase, now might be a good time to read my post + cheatsheet on that.

Take this completely reasonable and realistic scenario: you write an application for a dragon whose sole job it is to destroy villages:

class Trogdor 
{
    /** * @var Type */ 
    private $type; 
    public function __construct(Type $type = null) 
    { 
        $this->type = $type ?? Type::dragon(); 
    } 
    public function burninate(Village $village) 
    { 
        foreach ($village->getCottages() as $cottage) { 
            $cottage->burn(); 
        } 
    } 
    public function getType(): Type 
    { 
        return $this->type;
    } 
}

You did some refactoring and you’re a bunch of commits in, when suddenly you notice your dragon has stopped burning cottages. When and how did this happen? Let’s assume you know that last week everything was working fine.

Step 1: write a test

First, write a regression test that covers your expected behavior. At this point it will fail. Commit your test. Don’t worry about writing a pretty message.

class TrogdorTest extends TestCase
{
    public function testCottagesShouldBurn()
    {
        $application = new TrogdorApplication(new Trogdor(), Village::withCottages());
        $application->run();
        foreach ($village->getCottages() as $cottage) {
            self::assertTrue($cottage->isBurned());
        }
    }

}

Step 2: move the test

Using interactive rebase, move the commit back in time to the point everything was still working. In this example we’ll just assume that wasn’t so long ago, but bisect is especially useful in situations where you have many more commits to work through.

 git rebase -i HEAD~7

A file will open in your default text editor. Cut and paste the line to a place in time where the test will pass, then save and close the file:

pick 1b81fc5 Create Trogdor 
pick 51927a0 TROGDOR TEST
pick cf3ea2d Introduce dragon types
pick 6882ded He was a man
pick 95a11c7 I mean he was a... dragonman
pick 0f763a8 Or maybe he was just a... dragon
pick 8587b82 Add people to the cottages

This will reorder the commit. To check that the test now passes, you can check out the commit and run it:

git checkout 51927a0    // Replace this with the correct commit ID.
phpunit TrogdorTest.php // Check that this passes.
git checkout -          // Return to your HEAD.

Step 3: run bisect

If you’re unfamiliar with bisect, in a nutshell it’s a Git feature that helps you find a point in history efficiently. Instead of manually checking out and investigating commits one by one, bisect will automatically check out commits for you using a binary search strategy, making it much more efficient. Again, you might want to head over to my cheatsheet and jump to bisect for more context.

You can run bisect automatically by feeding it any script that will either succeed (exit code 0) or fail (anything else). That’s how tests succeed or fail, so your test is one such script. If you provide bisect a commit on which the test will pass (good), and one on which it will fail (bad), it will use binary search to automatically find the first point in time where it will fail:

git bisect start
git bisect good ea7753a                // Mark the passing commit as good
git bisect bad HEAD                    // Mark your HEAD as bad
git bisect run phpunit TrogdorTest.php

When bisect is finished running, it will present the commit that introduced the bug:

0f763a8374ecb38fdc02657f0502495227cb2d94 is the first bad commit

You can then exit bisect by running git bisect reset.

Step 4: fix the bug

Now that you know which commit introduced the bug, you can check it out and inspect the changes it introduced. In this case, it seems that you forgot to change a conditional that checks Trogdor’s type when you changed his default type from dragonMan to dragon:

class Application
{
    /**
     * @var Trogdor
     */
    private $trogdor;

    /**
     * @var Village
     */
    private $village;

    public function __construct(Trogdor $trogdor, Village $village)
    {
        $this->trogdor = $trogdor;
        $this->village = $village;
    }

    public function run()
    {
        if ($trogdor->getType()->isDragonMan()) { // <- This should check for `isDragon()`!
            $trogdor->burninate($village);
        }
    }
}

In your detached HEAD state, do what you need to do to make the test pass. Create a commit for the fix, and check out a new branch from there.

Step 5: apply the fix
Great, so now you’ve implemented a fix. But you’ll probably want to apply it to your original branch and commit it together with your test. So copy the commit ID of the fix, and check out your previous branch. Then, cherry pick your fix so that it’s on your branch:

git cherry-pick cc78654

Now, enter interactive rebase, move your test commit, and squash them together. Don’t forget to write a nice commit message describing the what and the why.

git rebase -i HEAD~8
pick 1b81fc5 Create Trogdor 
pick cf3ea2d Introduce dragon types
pick 6882ded He was a man
pick 95a11c7 I mean he was a... dragonman
pick 0f763a8 Or maybe he was just a... dragon
pick 8587b82 Add people to the cottages
pick cc78654 fix

All done! Of course, this was a simplified example. But in complex projects, it can be very hard to pinpoint exactly what went wrong. Especially if you’re looking at 20+ commits. You don’t want to go through them one by one, and with this strategy you won’t have to.

If you want to try it out yourself, clone this repo and check out a branch called 4_bisect. I’ve broken a test in the HEAD. The commit you’ll want to mark as good is 85ce6e4 Fix risky tests. Good luck 🙂