Evaluating the quality of test cases
“How much do we have to test?” – Every development department has to answer such a question. One answer to this question is the so-called code coverage. It expresses what proportion of the source code – statements, branches, paths, and conditions – is executed by test cases.
Code coverage – sometimes called code overlap – is expressed as a percentage. Without test cases, the code coverage is 0 percent. A high code coverage ideally reduces the development risk, but in the narrower sense says little about the quality of the tested source code, because a high coverage can also simply be a sign for many test cases.
There are several metrics for determining code coverage:
- Tested statements / statements. This metric is called statement coverage or C0-Coverage.
- Tested Branches / Branches. This metric is called Branch Coverage or C1-Coverage and refers to program loops.
- Tested paths / paths. This metric is called Path Coverage or C∞ Coverage and refers to program paths. Due to the combination of all loops and paths, which in many cases can be traversed multiple times, the number of paths is potentially infinite and a complete path coverage is therefore not realistic.
- Tested conditions/conditions. This metric is called predicate coverage of conditions, condition coverage or C1p coverage and addresses all predicates (i.e. all terms and partial terms).
In addition, various publications also mention decision coverage as an extended view of branch coverage and multicondition coverage as coverage of all condition combinations.
The metrics essentially determine the relationship of the test cases to the code or to individual code components and are regarded as a quality indicator for the test cases. Complicated code is difficult to test, a high code coverage is therefore a sign of good test cases. One criticism of the concept of code coverage is the neglect of actual test results. Whether a test was successful, whether and how many errors were found is not the focus of the code coverage.
In software development, it makes sense to define test cases on the basis of requirements. However, this does not ensure that all corresponding code parts are covered. Requirements can be implemented in different ways, so that there are correspondingly different reference points for tests. The requirement coverage is therefore not identical to the code coverage.
The principle of coverage can be applied to numerous artifacts. So it can give for example a draft, a module, an architecture, a data or a function coverage. They can be determined exactly like the code coverage with a simple relationship: Coverage = tested elements / specified elements.
You can find a blog post about the assessment of a Unit Test Coverage here »
“The expertise in software architectures, the expertise in software development and the very flexible way of working were ideal for us.“
“I have been reading your blog for a long time with great pleasure. Systematically, to the point and appealing.”