👾 Mutation testing on Go

One of the Developer's goals is to improve the healthiness of codebases. Fortunately, automated testing is a widely known practice to decrease production cycles and receive accelerated feedback. But, how we can check if our test suite are healthy? One of the techniques on automated testing to identify weak spots of dead/untested code is Mutation Testing.

---

What is Mutation Testing? 👾

Mutation testing or known as Mutation Analysis, involves to modifies an application in small ways programmatically and running against your test suite looking for weak spots of your code that has not test for a specific mutation.

Fundamental concepts involved over Mutation Testing:

• Mutants: a modified version that will test against a testing suite.
• Mutation Score: percentage of passed mutation/total mutation created total. 0 means all mutations created are alive 👾, so the tests are probably has not good coverage.

• Mutation operators/mutators: A mutator is an operation applied to the original code. Many types of mutators can be applied. It is important to check which mutators are available for your mutation tool.

• Equivalent Mutations: Are false-positives mutants. Sometimes, it will found a mutated version that has no practical changes. Often, they can mean a dead/useless code in the application.

When to use Mutation Testing?

It is an exciting alternative to the code coverage we see over different open source projects. One caveat about mutation testing that, for larger codebases, mutation testing can consume a significant amount of resources. So it can be decide to run it periodically through the codebase, not each push to your git remote repository.

---

Mutation Testing in Go

In practice, how mutation testing works? Let's see the follow code sample that will be the production code:

package main

func GreetingsByLocale(localestring) string {

   greetings := resolveGreetings(locale)

   if greetings == "" {
      return "Sorry, we didn't identified you"
   }

   return greetings
}

func resolveGreetings(localestring) string {
   if locale == "en" {
      return "Hello"
   }

   if locale == "pt" {
      return "Olá"
   }

   return ""
}

The example above, implements a very straightforward GreetingByLocale function, that returns the appropriated greetings by locale. Example: en , pt.

To running a mutation testing in Go, we have to install a mutation testing tool. For this article I've choose go-mutesting based what I was looking for at awesome-go.

How to install

Running the following command:

go get -t -v github.com/zimmski/go-mutesting/...

It will be ready to run the follow command without problem normally. If you have any problems, check it out documentation.

go-mutesting main.go

The command above, will produce:

-------- main.go   2021-06-06 14:33:00.000000000 +0200
+++ /var/folders/y0/z16t6n116rxgq599drwq_z31r3w4v8/T/go-mutesting-406451808/basics-ut/main.go.0 2021-06-06 14:41:17.000000000 +0200
@@ -5,7 +5,7 @@
        greetings := resolveGreetings(locale)

        if greetings == "" {
-               return "Sorry, we didn't identified you"
+
        }

        return greetings
@@ -21,4 +21,4 @@
        }

        return ""
-}
\ No newline at end of file
+}

...

FAIL "/var/folders/y0/z16t6n116rxgq599drwq_z31r3w4v8/T/go-mutesting-406451808/basics-ut/main.go.2" with checksum 7543b5d5e97c3a66dec555fb1c908957
The mutation score is 0.000000 (0 passed, 3 failed, 0 duplicated, 0 skipped, total is 3)

---

What information you should take care of?

Mutation Score: Percentage over passed mutation / mutation created total. 0 means all mutation created are alive, so the tests are probably not covering the whole usage of this package.

Mutation

        if greetings == "" {
-               return "Sorry, we didn't identified you"
+
        }

The output of mutation testing explains which it was modified that make the testing suite failed.

---

Fixing Mutations

You need to created the appropriated tests that covers the mutation. For it, you can implement something like:

func TestGreetingsByLocaleByDefault(t*testing.T) {
   greetings := GreetingsByLocale("")

   if greetings != "Sorry, we didn't identified you" {
t.Errorf("greeting() = %v; want 'Sorry, we didn't identified you'", greetings)
   }
}

Running again go-mutesting, you should see this output:

$ go-mutesting main.go

PASS "/var/folders/y0/z16t6n116rxgq599drwq_z31r3w4v8/T/go-mutesting-610805264/basics-ut/main.go.0" with checksum 73ab3954dddfcf60d062bf280b49e996
PASS "/var/folders/y0/z16t6n116rxgq599drwq_z31r3w4v8/T/go-mutesting-610805264/basics-ut/main.go.1" with checksum 0a29bb0da18ff5c339c726f678c8f4b9
PASS "/var/folders/y0/z16t6n116rxgq599drwq_z31r3w4v8/T/go-mutesting-610805264/basics-ut/main.go.2" with checksum 7543b5d5e97c3a66dec555fb1c908957
The mutation score is 1.000000 (3 passed, 0 failed, 0 duplicated, 0 skipped, total is 3)

Perfect! We fix all mutations found.

---

Going further

Mutation testing is very powerful technique to spot weak coverage points of your test suite. Some links you can found useful:

Explanation about mutation testing:
https://en.wikipedia.org/wiki/Mutation_testing

Mutation testing tools by language:
https://github.com/theofidry/awesome-mutation-testing

Mutation testing tool for Go:
https://github.com/zimmski/go-mutesting