What does a software code check cost

Code coverage serves as the primary metric for software testing completeness for most software developers.

This article gives you an overview of the degrees of coverage and gives tips on how to determine them.

1. What Code Coverage is

Code coverage is usually translated as Degree of coverage, Test coverage or Code coverage. This measure indicates the relationship between the tests actually carried out and the theoretically possible tests.

Depending on how the number of theoretically possible tests is determined, a distinction is made between statement coverage, branch coverage, decision coverage and other degrees of coverage.

a) Statement Coverage

The Instruction cover (English: Statement Coverage) corresponds to the statements run through during the test in relation to all statements. Examples of instructions would be value assignments for variables, case distinctions, loops and method calls. With many programming languages, it is good practice to write (only) one instruction per line of code.

example

In this example, if the method is called once with the values ​​(23, 80) and once with the values ​​(160, 80), not all lines are run through: red stands for lines that have not been run through and green for lines that have been run through. We'll talk about the yellow ones in a moment.

The instruction coverage is one of the weak degrees of coverage.

b) Branch Coverage

A stronger degree of coverage is that Branch cover. It corresponds to the ratio of the branches (also called edges) run through in the test. What a branch or an edge is can be understood using the example of a program graph (matching the code excerpt above).

If the function shown in Fig. 1 is called with the values ​​(23, 80), (160, 80), (180, 40) and (160, 60), all statements are run through, but not all branches. In our example, the branch that connects instructions 11 and 15 is missing (Fig. 2).

100% branch coverage requires 100% instruction coverage. The reverse is not true. So the branch coverage is the stronger dimension and not the instruction coverage.

c) Condition Coverage

Many conditions (if conditions, loop heads) consist of partial conditions. In our example, the condition in line 6 contains a total of four atomic partial conditions.

The branch coverage does not differentiate on the basis of which partial conditions a condition is evaluated as true or false. This achieves the Condition coverage. A distinction is made between single and multiple condition coverage.

Single condition coverage

The single-condition coverage indicates the proportion of atomic partial conditions that have assumed the value true at least once and the value false at least once.

100% single condition coverage does not result in 100% branch coverage, as the following example shows:

Overall condition = (subcondition_1 OR subcondition_2)

Test cases

  • Subcondition_1 = true, Subcondition_2 = false (not fulfilled)
  • Subcondition_1 = false, Subcondition_2 = true (fulfilled)

Although each of the two partial conditions was met once and not met once, the overall condition was always evaluated as "true". That is, the test never ran the "false branch".

Multiple condition coverage

With multiple condition coverage, the test cases are chosen so that all combinations of these conditions are tested. With four atomic partial conditions (as in the example above) at least 4 is required2 = 16 test cases. For 5 partial conditions there would be 32 test cases, since the number 2n is, where n is the number of partial conditions.

d) Further code coverage measures

Additional levels of coverage include loop coverage and path coverage.

Additional information

Please also note the article on combinatorial testing, which introduces additional "coverage levels" that are even suitable for black box tests.

2. Regulatory requirements for code coverage

There are normative requirements in many regulated industries such as the aircraft industry or medical technology. You will find this presented here using the example of medical technology.

a) Europe

After all, IEC 62304 (version 1.1) knows the term code coverage. However, the standard does not require this metric to be recorded at all. In the non-normative part, however, it recommends increasing the code coverage.

An auditor cannot certify non-conformity if you do not determine the code coverage. He or she can, however, ask for a reason why the definition and verification of the acceptance criteria for a software unit are considered to be sufficiently complete.

b) USA, FDA

The requirements

The FDA goes on to code coverage in the Guidance Document General Principles of Software Validation a. She writes:

"Measures such as [...] testing coverage [...] are all used to develop an acceptable level of confidence before shipping the product."

Discussing the various levels of coverage, the FDA notes:

"It [Decision Coverage] is considered to be a minimum level of coverage for most software products, but decision coverage alone is insufficient for high-integrity applications."

The FDA understands by "Coverage" means 100% coverage - regardless of the type of coverage.

The reality

In our many years of consulting practice, we have not met any medical device manufacturer who has approved products in the USA and achieved 100% code coverage. We have very rarely seen cases where an FDA inspector brought this up.

Read below how you can proceed to avoid difficulties with FDA inspections in this regard.

3. Critical evaluation of code coverage

a) Advantages

Without "sufficient" testing, a manufacturer cannot assume that its software is sufficiently free of errors. A necessary (but not sufficient) prerequisite for assuming that testing is “sufficient” is that the code is (completely) run through during testing.

The degrees of coverage are (thanks to tools) an easy-to-collect, quantitative metric for the completeness of tests. This enables developers, managers and auditors to identify which parts of the software need to be tested more intensively.

b) Disadvantages

When determining code coverage, consider the following:

  1. Source code required
    The source code is a prerequisite. Libraries and compiled artifacts can be tested as black boxes, but the degree of coverage cannot usually be determined.
  2. Not fully testable
    Parts of the code, such as instructions that are very close to the hardware, also resist testing and thus determining the code coverage.
  3. No proof of correctness
    Even a complete test coverage of 100% Decision Coverage does not constitute a proof of correctness of the software. For this all possible input values ​​would have to be tested, which is combinatorial in almost all cases impossible.
  4. expenditure
    Setting up the tools to determine the code coverage means a (one-time) effort. Writing the test as well.
  5. Timing behavior
    Tools change the source code e.g. by counting insert. This instrumentation can change the timing behavior and thereby possibly mask errors that would be visible without the instrumentation.
  6. Tool support
    The tools available on the market do not allow all of the above variants of test coverage to be determined (which is not always necessary).

4. Tips for Determining Code Coverage

A software development that claims professionalism will not do without determining the code coverages. So here are a few tips.

a) Integration in the development process

Specify in the corresponding procedural instructions, work instructions or in the software development plan which variant of the code coverage must be achieved to which degree. Also demand that the unit tests are developed parallel to the code and not just at the end of the software development.

b) Dealing with old code

So that you do not become overwhelmed with old code, you can demand that (only) every new and every changed method be tested and the code coverage determined.

Alternatively, you can start with very low goals and gradually increase them over time.

c) Risk-based approach

If you are working with less than 100%, then determine the degree of coverage for different components differently, e.g. depending on

  • the safety class (i.e. the maximum damage that an error would cause),
  • the reliability of the component or
  • the complexity of the component, which you quantify using the McCabe measure.

In the latter case, it could be required, for example, that the branch coverage must be 100% for all methods with a cyclomatic complexity (McCabe measure) of eight or more. For methods with a complexity of three to seven, a branch coverage of over 50% must be achieved.

d) refactoring

A common reason for low coverage levels is that the code cannot be tested at all. This is a strong indicator for a bad architecture, e.g. for a missing hardware abstraction layer (HAL) or for components that are not functionally formed. Use these low test coverage as an indicator of where refactoring is needed.

e) tools

Don't forget to add the code coverage tool to your list of tools to validate. This requires that you keep the tool including the configuration files under version control, if necessary on all developer machines.

Here is a list of tools to help you determine coverage levels.

f) FDA inspections

When an FDA inspector asks why you achieved test coverage below 100%, be prepared with argumentation. The following arguments can help you:

  • A test coverage (e.g. branch coverage) of 100% is usually not possible.
  • They followed the FDA's recommendation for a risk-based approach and indeed tested critical methods with the required degree of coverage.
  • You flank your analytical quality assurance with further measures such as code review and static code analysis.
  • The increasing test coverage shows that you are continuously improving.
Additional information

Tip: You can also use the coverage levels to estimate the probability of software errors. Read more about this here.