Understanding Software Test Coverage Criteria Step by Step: From Line Coverage to MC/DC

Software testing methods in development often have similar names, which can be quite confusing. Understanding their differences helps avoid confusion when implementing tests and makes it easier to explain them to non-testing team members.

This time, let’s go through five basic coverage criteria in software testing, from fundamental to advanced, step by step. Each of these coverage criteria was developed to address the shortcomings of the previous ones.

• Line Coverage helps identify unexecuted code but doesn’t ensure sufficient testing of conditional branches.
• Branch Coverage tests both true and false outcomes of conditions but doesn’t consider the impact of individual conditions.
• Condition Coverage checks each condition’s true/false outcomes, but short-circuit evaluation can lead to gaps in coverage.
• Multiple Condition Coverage tests all possible combinations, but the number of test cases can explode.
• MC/DC (Modified Condition/Decision Coverage) minimizes test cases while ensuring each condition's independence is tested, leading to high-quality test results with fewer tests.

Each of these coverage criteria was designed to compensate for the shortcomings of the previous approach, ensuring better overall test effectiveness.

Line (Statement) Coverage

This one is straightforward. It simply measures how much of the code was executed by the test. The formula is number of executed lines / total lines of code. At first, I wondered if this had any real significance, but it turns out that line coverage is useful in the early stages of development to identify unexecuted parts of the code. Especially for humans, it's easy to understand when test tools highlight executed and unexecuted lines in green and red during source code reviews. However, since line coverage alone is not enough to verify software quality, it is usually combined with branch coverage.

Branch (Decision) Coverage

This measures coverage by ensuring that each conditional branch has been executed at least once for both True and False outcomes. The key point here is that the focus is on results.

For example, given the condition if (A && B), there are four possible combinations, but branch coverage requires only two test cases: No.1 and one from [No.2, No.3, or No.4], since that ensures both True and False outcomes.

No	A	B	Result
1	True	True	True
2	True	False	False
3	False	True	False
4	False	False	False

When there are multiple conditions, each condition's effect must be checked independently. Consider the following pseudo-code:

if A==True {
doSomething1();
}
if B==True {
doSomething2();
}
if C==True {
doSomething3();
}

This code contains three independent branches. To satisfy branch coverage, each conditional branch must be executed for both True and False outcomes. At first glance, it might seem that just two cases, "TTT" and "FFF," would be enough, but that would fail to confirm independent effects of each condition. Without this concept of independence, unnecessary tests could be added, or some test cases might be missing.

For example, if the test cases were only "TTT" and "FFF," they wouldn’t separately verify the impact of changing only A or only B. The essence of branch coverage is to confirm the independent behavior of each branch.

Therefore, the following four test cases are necessary:

Case	A	B	C	if execution paths
1	T	T	T	All if statements are executed
2	F	T	T	if(A) is skipped
3	T	F	T	if(B) is skipped
4	T	T	F	if(C) is skipped

Condition Coverage

While branch coverage focuses on whether conditional statements like if and else are executed at least once, condition coverage focuses on whether changes in each part of the condition affect the outcome. It ensures that each condition is executed at least once as both True and False.

Let's revisit the example from branch coverage. Given if(A && B), there are four possible combinations. In condition coverage, we need a test pair that ensures A and B each take both True and False values. This can be achieved with either [No.1, No.4] or [No.2, No.3], reducing the required test cases to two.

No	A	B	Result
1	True	True	True
2	True	False	False
3	False	True	False
4	False	False	False

A key issue here is short-circuit evaluation. For instance, in if(A && B), if A is False, B is never evaluated. This is problematic because:

1. B is not actually tested in execution, making the test incomplete.
2. If B has side effects (e.g., function calls or variable modifications), skipping its evaluation can cause behavioral differences that go unnoticed.
3. Bugs related to B might remain hidden.

Additionally, condition coverage does not ensure full branch coverage. This limitation leads to the need for more advanced coverage criteria.

Multiple Condition Coverage

Condition coverage alone can leave gaps in test coverage. To avoid this, multiple condition coverage tests all possible combinations of conditions. This is a stricter criterion than condition coverage and eliminates gaps, but it comes with significant drawbacks:

• Exponential growth in test cases

  • With N conditions, 2^N test cases are required.
  • More conditions mean an explosion in test cases, making it impractical.

• Redundant test cases

  • It includes cases with identical outcomes.
  • Many tests might be functionally meaningless.

• Inefficiency due to short-circuit evaluation

  • In if(A || B), if A is True, B is never evaluated.
  • This leads to unnecessary tests for conditions that are never executed.

In practice, minimizing test cases while ensuring proper condition testing is key. Condition coverage is often sufficient, but for high-reliability systems such as aviation and medical software, MC/DC (Modified Condition/Decision Coverage) is used.

MC/DC (Modified Condition/Decision Coverage)

MC/DC ensures that each condition independently affects the outcome, covering all necessary condition changes with the minimum number of test cases.

To satisfy MC/DC, the following four conditions must be met:

1. Each condition must take both True and False at least once.
2. The result must take both True and False at least once.
3. Each condition must independently affect the overall result.
4. This must be achieved with the minimum number of test cases.

This might sound abstract, so let’s go through a concrete example. Suppose we are offering apple juice, and the conditions are whether we have a cup, juice, and ice. To serve apple juice, a cup and juice are absolutely necessary, while ice is optional. The full set of possible combinations is as follows:

Case	cup	juice	ice	result
1	T	T	T	OK
2	T	T	F	OK
3	T	F	F	NG
4	F	T	T	NG
5	F	F	T	NG
6	F	T	F	NG
7	T	F	T	NG
8	F	F	F	NG

MC/DC focuses only on conditions that independently affect the result. Since ice does not affect the result, it is excluded from testing. We then check whether changing cup or juice individually alters the result.

• Case 1 (T,T,T) is chosen to represent a valid (OK) result.
• Case 4 (F,T,T) tests the effect of removing the cup.
• Case 7 (T,F,T) tests the effect of removing the juice.

Case	cup	juice	ice	result
1	T	T	T	OK
4	F	T	T	NG (cup's effect)
7	T	F	T	NG (juice's effect)

With these selected test cases, MC/DC criteria are satisfied. Since MC/DC ensures that each condition independently affects the outcome, it optimizes test cases while considering short-circuit evaluation, making it a highly efficient testing method.

Understanding the Strengths and Weaknesses

For those new to software testing, the numerous similar-sounding test names and definitions can be confusing. Many explanations assume that each condition independently affects the outcome, which makes understanding even harder.

When creating real-world test cases, balancing time, cost, and acceptable risk is crucial. Having a solid grasp of what each coverage criterion aims to achieve can be valuable for all stakeholders involved in testing.

References

• Qt: Line Coverage
• What is MC/DC?
• Measuring Code Coverage: Guide to Effective Testing