You’ve all heard of the Test Pyramid — E2E Tests, Integration Tests, and Unit Tests. It’s popular. But it has BIG problems — misinterpretations of test types and missing test types. It’s dangerous.
The Original Test Pyramid
The Test Pyramid was introduced by Mike Cohn in 2009 in his book Succeeding with Agile: Software Development Using Scrum. He has the following levels:
- UI Tests
- Service Tests
- Unit Tests
UI Tests
Mike Cohn indicates that the minority of the tests should be UI tests. He notes their disadvantages — that they are: “Brittle”, “Expensive to write” and “Time consuming”.
“Automated user interface testing is placed at the top of the test automation pyramid because we want to do as little of it as possible.” — Mike Cohn, Succeeding with Agile
Service Tests
He places Service Tests in the middle, to avoid the situation that we have too many UI tests, instead we want to test at the layer below the UI.
“Service-level testing is about testing the services of an application separately from its user interface. So instead of running a dozen or so multiplication test cases through the calculator’s user interface, we instead perform those tests at the service level.” — Mike Cohn, Succeeding with Agile
However, it should also be noted that he isn’t too precise regarding the definition of service tests, that it doesn’t just mean SOA but can refer to anything that’s transforming inputs into outputs:
“Although I refer to the middle layer of the test automation pyramid as the service layer, I am not restricting us to using only a service-oriented architecture. All applications are made up of various services. In the way I’m using it, a service is something the application does in response to some input or set of inputs.” — Mike Cohn, Succeeding with Agile
Service Tests are often forgotten, so he wrote a blog post about the forgotten service level testing.
Unit Tests
Mike Cohn indicates that the majority of our tests should be unit tests. Unit tests are useful due to the fast feedback they provide, that we can easily localize regression bugs within code:
“At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid. Automated unit tests are wonderful because they give specific data to a programmer — there is a bug and it’s on line 47.” — Mike Cohn, Succeeding with Agile
The Popularized Test Pyramid
If you do a Google search for Test Pyramid images, most likely you’ll end up with an image that looks like this, with these layers:
- E2E Tests
- Integration Tests
- Unit Tests
That’s THE way to do automated testing, right?
Wrong.
Using the “Popular” version of the Test Pyramid has led to the following problems:
- Overuse of E2E tests, often reaching the Inverted Test Pyramid
- No consensus on the definition of Integration Tests, and over-reliance on large scoped integration tests makes it hard to maintain
- No consensus on the definition of Unit Tests, and writing unit tests that are structurally coupled to code with over-mocking, thus fragile when refactoring
- The Test Pyramid does not include Acceptance Tests needed for testing Acceptance Criteria, but instead it relies on fragile E2E Tests for that
- The Test Pyramid doesn’t provide us guidance for testing microservices, specifically it doesn’t even mention Component Tests & Contract Tests which are crucial for microservice testing
To dive deeper into the problems of the old Test Pyramid, read the full article on Optivem Journal.
Top comments (2)
I think that the worst thing is when developers write fragile unit tests and then give up on testing, because the bad the way they're writing them requires more maintenance than necessary.
Exactly, there are so many developers who had a bad experience with unit testing, because they spend more time maintaining those tests than writing code, and it "freezes" the design -if they did refactoring of the UML class diagram, many tests would break, so then they avoid that refactoring, and the design just deteriorates.