Unmasking Hidden Test Pitfalls in Go: Avoiding False Positives

The nightmare in testing would be false positive. “Every thing is passing! Amazing!” until at an unknown time in the future all the mines explode together and blow your team to hell.

There are many reasons that tests could fail silently.

Today, I’m going to talk about one very basic reason: don’t know which are tests.

Why you don’t know which are tests?

Most people join a Go project half-way. Most people learn a language by using it in a real life.

Therefore, when someone had set up the project with a test framework like testify, you would most likely think methods like the following are tests.

func (suite *ExampleTestSuite) TestExample() {
    suite.Equal(5, suite.VariableThatShouldStartAtFive)
}

You then add another method like TestAnotherCase and find it works. You think you’re crystal clear about what are tests.

Test has different meanings in different frameworks

A “test” you’re talking may not be the same test a Go package is talking.

From the built in testing package, a test is any function of the form

func TestXxx(*testing.T)

Of course, since the built-in testing package has limited features, most projects are using testify/suite or other similar third-party package as their test framework. What is a test from the testify/suite's point of view?

add any methods that start with "Test" to add tests

See, we have two different definitions of a test.

The trouble starts when using a third-party testing tool

When using some tools like mockery, you’ll read the following

you won't have to worry about forgetting the AssertExpectations method call anymore … The AssertExpectations method is registered to be called at the end of the tests

Great! “So I only need to create a mock and the package will notify me when expected behaviors happen”.

That’s where the trap is.

When mockery says at the end of the tests, it actually means the definition from testing , not the definition from testify/suite.

So when you have the following code, you’ll see both TestA and TestB pass even they should both fail because the mock setup in TestA is used in TestB.

package mockandsubtest

import (
    "fmt"
    "testing"

    "github.com/stretchr/testify/suite"
)

// Prod code
type ExternalService interface {
    Work()
}

type Server struct {
    externalService ExternalService
}

func NewServer(externalService ExternalService) *Server {
    return &Server{
        externalService: externalService,
    }
}

// Test code
type ServerSuite struct {
    suite.Suite
    ExternalService *MockExternalService
    Server
}

func TestServerSuite(t *testing.T) {
    suite.Run(t, &ServerSuite{})
}

// Run before all test cases
func (s *ServerSuite) SetupSuite() {
    s.ExternalService = NewMockExternalService(s.T())
    s.Server = Server{externalService: s.ExternalService}
}

// In this test, Work is set up to be called once but not called
func (s *ServerSuite) TestA() {
    fmt.Println("TestA is running")
    s.ExternalService.EXPECT().Work().Times(1)
}

// In this test, Work is called once unexpectedly
func (s *ServerSuite) TestB() {
    fmt.Println("TestB is running")
    s.Server.externalService.Work()
}

The result of running the above code is

TestA is running
TestB is running
PASS

Explanation

It turns out only TestServerSuite is considered as a test from testing and mockery's point of view. That’s why the AssertExpectations is called at the end of TestServerSuite , even though TestA and TestB are internally executed by testify/suite.

From mockery's point of view, s.ExternalService is expected to be called once and actually called once in the lifecycle of TestServerSuite. So the expectation is fulfilled.

How to mitigate?

There are two ways to bridge the gap between testify/suite and testing.

The first way is creating a new mock before each test method like the following.

func (s *ServerSuite) SetupTest() {
    s.ExternalService = NewMockExternalService(s.T())
}

Sometimes, it’s not practical in your project due to many reasons like setting up a server instance for each test case is too expensive. Then you can try the other direction, which is manually asserting after each test.

The second one is adding a call of AssertExpectations at the end of each test method. For example, call AssertExpectations in TearDownTest , which is executed after each test method.

func (s *ServerSuite) TearDownTest() {
    s.ExternalService.AssertExpectations(s.T())
}