This chapter describes the code analysis done by Squish Coco in detail.
In order to analyze the code coverage of e.g. a test suite, it is necessary to compile a version of the application in which statements are inserted that record the execution of each part of the source code. The generation of such a modified version of a program is called instrumentation.
Several kinds of code coverage are possible with Squish Coco. The most common are:
Here each condition counts twice, which may result in a large number of possible outcomes in a complex decision.
The coverage of a program is the number of executed statement blocks and of conditions that were tested independently divided by the number of statement blocks and conditions in the program.
Other, simpler coverage metrics are also supported.
Relevance for safety standards: IEC 61508 highly recommends 100% structural test coverage of entry points (i.e. functions) for SIL 1, 2, 3 and 4.
Line coverage is however an unstable coverage measure since it depends strongly on the way the code is formatted (see Chapter 5.6). We do not recommend line coverage.
In many safety standards, specific coverage levels are required.
The following table shows which coverage levels are required by these safety standards.
IEC 61508 ISO 26262 EN 50128 DO 178 Function Coverage (Entry Points) X Statement Block Coverage X X X X Decision Coverage (Branch Coverage) X X X X Modified Condition/Decision Coverage X X X X Multiple Condition Coverage X
Depending on the security levels the coverage requirement is either just recommended, highly recommended or required. More detailed information can be found at the end of the descriptions of the coverage metrics in the following sections.
In this section we describe the coverage metrics supported by Squish Coco in greater detail; the code that is inserted during the instrumentation process is described in more detail in Section 43.
In the next sections, we will use the following function to illustrate the coverage metrics and the instrumentation process. It is written in C++; coverage with other languages it is similar.
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n");
13 }
Figure 5.1: Example function for simple coverage metrics
The most elementary kind of instrumentation records the statements in a program that are executed when it runs. It is however not necessary to record the execution of every statement to get this information. If several statements form a sequence, it is enough to record how often the last statement is executed, since they all form a block that is either executed as a whole or not at all. Squish Cocotherefore inserts instrumentation statements only at the end of each block, and the resulting coverage metric is called Statement Block Coverage.
In the following listing, the instrumented statements are underlined:
1 void foo()
2 {
3 bool found=false; // not instrumented
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n"); // not instrumented
13 }
Figure 5.2: Code coverage at statement block level
One can see that not all lines are instrumented. The conditions of the if and for statements are not covered since they are no statements themselves. Some other statements need not to be instrumented because they are part of a block. In the example, these are the statements at lines 3 and 12: they belong to the block that begins at line 2 and ends with line 13. It is therefore enough to put an instrumentation point at line 13 to verify that all of them were executed.
ISO 26262 highly recommends Statement Coverage for ASIL A and B. ASIL C and D recommend it as well but the more strict Branch Coverage and Modified Condition/Decision Coverage are highly recommended instead.
A more detailed coverage metric also records the values of the Boolean conditions in branch and loop statements, like if, while, for, etc. To reach full coverage, the decision in such a construct must have evaluated to true and to false at least once. This kind of code coverage is called Decision Coverage or sometimes Branch Coverage.
Decision coverage also handles switch statements. In them, full coverage is reached if every case label in the code was reached at least once during the execution of the program.
Note that Decision coverage also include the coverage of statements, as in Statement Block Coverage. In the following listing, the conditions instrumented for Decision Coverage are displayed with a gray background. The instrumented statements are underlined as before. They are the same as in the previous listing.
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n");
13 }
Figure 5.3: Code coverage at decision level
ISO 26262 recommends Branch Coverage for ASIL A and highly recommends the method for ASIL B, C and D.
EN 50128 recommends Branch Coverage for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.
DO 178 mandates that Decision Coverage should be satisfied with independence for Software Level A and B.
IEC 61508 recommends Branch Coverage for SIL 1 and 2 and highly recommends this level for SIL 3 and 4.
Often one wants to analyze the Boolean decision in the if, while, for, etc. statements in greater detail. For this one can use Condition Coverage.
In this coverage metric, each decision is decomposed into simpler
statements (or conditions) that are connected by Boolean
operators like ’~’, ’||’ and ’&&’.
Then for full coverage of the decision, each of the conditions must
evaluate to true and to false when the program is
executed.
Note that ’||’ and ’&&’ are shortcut operators
and that therefore in a complex decision, not all conditions are
always executed. An example is the code fragment
‘if (a || b) return 0;’. In it, the conditions are the
variables a and b. To get them both evaluate
to true and false, one has to run the code fragment
once with a == true and b arbitrarily (since
it is not evaluated), and also two times with a == false:
in one run b is set to true, in the other run it
is set to false.
This is in contrast to the situation with decision coverage, where it
is enough for full coverage that the whole decision,
’a || b’, evalutes to true and to false.
With Condition Coverage, our example function is instrumented in the following way. As before, the instrumented Boolean expressions are gray:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i<100) && (!found); ++i) {
5 if (i==50)
6 break;
7 if (i==20)
8 found=true;
9 if (i==30)
10 found=true;
11 }
12 printf("foo\n");
13 }
Figure 5.4: Code coverage at condition level
We can see here that the decision of the for statement has been split into two separately instrumented conditions.
CoverageScanner also instruments the assignment of Boolean expressions to a variable. Constant or static expressions are not instrumented. This is because their values are computed at compile time or only once during program initialization, so full coverage cannot be reached anyway.
In the following listing, the instrumented Boolean expressions are again gray:
1 void foo()
2 {
3 static bool a = x && y;
4 bool b = x && y;
5 bool c = b;
6 int d = b ? 0 : 1;
7 }
Figure 5.5: Code coverage of assignments
We see that the statements at lines 3 and 5 are not instrumented – the first is evaluated at compile time and the second does not involve a Boolean operation.
Since Squish Coco works on a syntactic level, it cannot distinguish between expressions that involve only Booleans and those with other data types that support Boolean operators – all of them are instrumented.
The instrumentation tries to have no effect on the program, but
sometimes this is not possible. One example are older versions of
C#. They need to be instrumented with the option
--cs-no-csharp-dynamic: Then the instrumented code converts the
operands of a Boolean operator like ’||’ and
’&&’ first to Boolean, before the operator is applied.
(Short circuit evaluation still works, of course.) This is different
from the way in which an uninstrumented program would do it.
In C# programs that are compiled with Microsoft® Visual Studio® versions before 2010, it is therefore necessary that both cast operators true and false are defined for the objects that are arguments of Boolean operators. For newer C# versions, the default settings of the CoverageScanner can be used and the instrumented code does not have this problem.
The properties of Multiple Condition Coverage and MC/DC become only visible when a program contains a complex decision with many conditions. Our previous example therefore cannot be used anymore. Instead, we will use the following program, which replaces some letters in a string with underscores: those between ‘a’ and ‘e’ (inclusive) in the alphabet, and those that follow the full stop at the end of the sentence. (There is no character after the full stop in the example text, but that is on purpose: We need some conditions that fail.)
1 #include <stdio.h>
2
3 int main()
4 {
5 char text[] = "The quick brown dog jumps over the lazy fox.";
6
7 for (char *p = text; *p; p++) {
8 if ((*p >= 'a' && *p <= 'e') || (p != text && *(p - 1) == '.'))
9 *p = '_';
10 }
11
12 puts(text);
13 return 0;
14 }
Figure 5.6: Example program for Multiple Condition Coverage and MC/DC
We will concentrate on the complex condition in line 8. When the coverage of this program is shown in the CoverageBrowser, one can move the mouse over that line and see the following table, which describes the coverage data. (Or one can click on the line and see a similar table in the "Explanation" window.)
*p >= ’a’ *p <= ’e’ p != text *(p - 1) == ’.’ Description TRUE FALSE TRUE TRUE never executed FALSE TRUE TRUE never executed TRUE FALSE TRUE FALSE executed 27 times by 1 test FALSE TRUE FALSE executed 9 times by 1 test TRUE FALSE FALSE never executed FALSE FALSE executed 1 time by 1 test TRUE TRUE executed 7 times by 1 test
Figure 5.7: Multiple Condition Coverage table for line 8
In this table, each line contains a combination of condition results. The first four columns contain the results of a single condition. Since C uses shortcut operators, not all conditions are shown executed in each row. Rows that were executed have a green background, rows that were not have a background in red.
From the current table we see that three combinations of conditions
were not executed and are missing from full condition coverage. To
execute e.g. the condition combination in the second red line, it
would be necessary to make the condition *p >= 'a' false
and the conditions p != text and
*(p - 1) == ' ' true. To reach this, one could put a
capital letter (for the first condition) behind the full stop (for the
two others) in text.
EN 50128 recommends MCC (or Modified Condition/Decision Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.
Modified Condition/Decision Coverage (MC/DC) is a variant of Multiple Condition Coverage that requires fewer tests. Its goal is to make sure that for each condition in a complex decision there are two executions that differ only in the result of that condition and the condition result.
The condition table of MC/DC is a more complex version of the table for Multiple Condition Coverage:
*p >= ’a’ *p <= ’e’ p != text *(p - 1) == ’.’ Decision Description TRUE FALSE TRUE TRUE TRUE never executed FALSE TRUE TRUE TRUE never executed TRUE FALSE TRUE FALSE FALSE executed 27 times by 1 test FALSE TRUE FALSE FALSE executed 9 times by 1 test FALSE FALSE FALSE executed 1 time by 1 test TRUE TRUE TRUE executed 7 times by 1 test TRUE FALSE FALSE FALSE never executed
Figure 5.8: MC/DC table for line 8
Its large-scale structure is the same as with Multiple Condition Coverage, but there are some new features:
To see which rows contributed to the coverage of a covered condition, search in the column that belongs to this condition for dark green fields. Any two rows with a dark green field contribute together to the coverage of the condition if one of the dark green fields has a TRUE entry and the other one a FALSE entry.
In the example, we see that the first condition,
‘*p >= 'a'’, is covered in two different ways: by the
second green row together with the fourth green row, and also by the
third and fourth green row together. The second condition,
‘*p <= 'e'’, is covered by the first and fourth green
row together. One can also see that these two rows only differ in
the second column and in the decision; the other columns contain the
same results or at least one empty field.
You may have noted that the table is sorted in a different way from that one for Multiple Condition Coverage: First there are the red rows, then the green rows and then the light red ones. This is always the case.
ISO 26262 recommends MC/DC for ASIL A, B, C and highly recommends this method for ASIL D.
EN 50128 recommends MC/DC (or Multiple Condition Coverage) for SIL 1 and 2. For SIL 3 and 4 this level is even highly recommended.
DO 178 mandates that MC/DC should be satisfied with independence for Software Level A.
IEC 61508 recommends MC/DC for SIL 1, 2 and 3 and highly recommends this level for SIL 4.
The CoverageBrowser is a graphical user interface program to display the analyzed results of the instrumentation. It uses a color coding to indicate the status of the statements. In this manual we use the same colors as in the program.
There are two kinds of statements in an instrumented program. Some contain an instrumentation point, i.e. a piece of code inserted by Squish Coco which increments a counter when it is executed. If a line contains an instrumentation point, it is shown on a dark-colored background by the CoverageBrowser and in the HTML reports.
For reasons of efficiency, not all statements contain instrumentation points. If a line does not contain an instrumentation point but its coverage status can be inferred from other statements, it is shown on a light-colored background.
The resulting color scheme is then:
while and
if or as the right side of an assignment to a boolean
variable.Boolean expressions are always instrumented, therefore a light orange background is not necessary.
The output for our example function illustrates this:
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found); ++i)
5 {
6 if (i==50)
7 break;
8 if (i==20)
9 found=true;
10 if (i==30)
11 found=true;
12 }
13 printf("foo\n");
14 }
Figure 5.9: Code coverage output
The for statement in line 4 is shown in orange since it
contains the expression ’i<100’, which is only partially
executed: it always takes the value true when the function
is executed. The expressions ’i==50’ and
’i==30’ were also partially executed, and we can see from
the way the statements ’break;’ and
’found=true;’ are colored that in both cases the condition
evaluates to false. (In CoverageBrowser, the value of the
condition is also shown in a tooltip.)
Lines 3, 5, and 13 are not directly included in the coverage measurements and are therefore shown on a light background. Their coverage states are inferred by Coco from other statements that were (or were not) executed later. In our example, lines 3 and 13 must have been executed because the closing brace in line 14 has been executed, and line 3 has been executed because of line 12. Therefore all the lines with light background are shown in green.
The insertion of code during the instrumentation increases the code size and also affects performance of the instrumented application. It will use more memory and run slower.
For non-conditional expressions, the instrumentation code is only a write instruction to a counter at a fixed memory location. However, for conditional expressions, more detailed analysis is needed and this is more computationally expensive.
Overall, an instrumented application will be 60% to 90% larger and will run 10% to 30% slower.
Some developers write more lines of code than others simply by using a particular coding style—for example by putting opening braces on a line of their own rather than on the same line as an if statement. Therefore Squish Coco uses by default a coverage metric that is not susceptible to such differences in coding style. Its calculations are based on the number of executed instrumented instructions compared with the total number of instrumented instructions.
Every instrumented simple statement (like return,
break, the last instruction of a function, etc.) is recorded
by a single instrumentation counter. A fully instrumented condition in
an IF...THEN...ENDIF block uses has two instrumentation
counters: one for the true case and one for the false case. If the
code is only partially instrumented, only one condition is recorded
(either the false case or the true case).
The statistics itself depends of the type of instrumentation. It is in general not possible to compare code coverage at statement block level with that at decision level: having 80% coverage at decision level does not tell us anything about the coverage at statement block level, which could be bigger or smaller. We can only be sure that reaching 100% coverage is more difficult with condition coverage than with decision or statement block coverage.
In our example code, the coverage is between 60% and 75%. The following table shows the details for each instrumentation type.
In the examples below, the details of the calculations are displayed with subscripts. The first number in a subscript shows how many instrumentated statements were executed; the second is the number of instrumented statements in total.
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found); ++i)
5 {
6 if (i==50)
7 break;0/1 [not executed]
8 if (i==20)
9 found=true;1/1 [executed]
10 if (i==30)
11 found=true;0/1 [not executed]
12 }1/1 [executed]
13 printf("foo\n");
14 }1/1 [executed]
Figure 5.10: Code coverage statistics for full statement block instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found)2/2 [was false and true]; ++i)
5 {
6 if (i==50)1/2 [was false but not true]
7 break;0/1 [not executed]
8 if (i==20)2/2 [was false and true]
9 found=true;1/1 [executed]
10 if (i==30)1/2 [was false but not true]
11 found=true;0/1 [not executed]
12 }1/1 [executed]
13 printf("foo\n");
14 }1/1 [executed]
Figure 5.11: Code coverage statistics for full decision level instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100) && (!found)1/1 [was false]; ++i)
5 {
6 if (i==50)1/1 [was false]
7 break;0/1 [not executed]
8 if (i==20)1/1 [was false]
9 found=true;1/1 [executed]
10 if (i==30)1/1 [was false]
11 found=true;0/1 [not executed]
12 }1/1 [executed]
13 printf("foo\n");
14 }1/1 [executed]
Figure 5.12: Code coverage statistics for partial decision level instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100)1/2 [was true but not false] && (!found)2/2 [was false and true]; ++i)
5 {
6 if (i==50)1/2 [was false but not true]
7 break;0/1 [not executed]
8 if (i==20)2/2 [was false and true]
9 found=true;1/1 [executed]
10 if (i==30)1/2 [was false but not true]
11 found=true;0/1 [not executed]
12 }1/1 [executed]
13 printf("foo\n");
14 }1/1 [executed]
Figure 5.13: Code coverage statistics for full condition level instrumentation
1 void foo()
2 {
3 bool found=false;
4 for (int i=0; (i < 100)1/2 [was true but not false] && (!found)2/2 [was false and true]; ++i)
5 {
6 if (i==50)1/1 [was false]
7 break;0/1 [not executed]
8 if (i==20)1/1 [was false]
9 found=true;1/1 [executed]
10 if (i==30)1/1 [was false]
11 found=true;0/1 [not executed]
12 }1/1 [executed]
13 printf("foo\n");
14 }1/1 [executed]
Figure 5.14: Code coverage statistics for partial condition level instrumentation
Line coverage is a natural metric which allows easily to see which lines of code are executed. But it is less accurate than the instrumentation at statement block level and its results rely on the developer’s coding style.
The following example illustrates this problem:
Executing it would produce the following result:
This execution correspond to a coverage of 33%.
Since the first line of main contains two executed
statements, splitting it in two increases the number of executed
lines, and so the test coverage. So if we reformat the main function
as follows,
the execution of this code results in a code coverage of 66%:
Another way to increase the coverage by reformatting is to hide an uncovered statement behind an executed one. To do it it is only necessary to write the whole code of this main function in one line:
This code has a line coverage of 100%:
This small example illustrate how the result depends on how the source code is formatted. Therefore, Squish Coco provides the line coverage as additional measurement to the decision and the condition coverage and so does not allow that a source code is only instrumented at line level.
This chapter discusses the various testing strategies that can be carried out so as to make the best use of the Squish Coco tools.
If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executing just portions of its code, or executing code just once, is insufficient.
To address this issue the Squish Coco tools provide two possible approaches:
Squish Coco can be adapted to fit in with many different testing strategies, from unit tests during early development to complete product validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverage results at each stage of testing and so provides a precise overview of the software’s overall quality.
White box testing (a.k.a. clear box testing, glass box testing or structural testing) uses an internal perspective of the system to design test cases based on internal structure. It requires programming skills to identify all paths through the software. The tester chooses test case inputs to exercise paths through the code and determines the appropriate outputs.Definition from Wikipedia
Interactive white box tests are easy to do using Squish Coco: once the application has been compiled with CoverageScanner, the test engineer can execute the it, and after each execution they can import the execution report into CoverageBrowserfor analysis.
Black box testing takes an external perspective of the test object to derive test cases. These tests can be functional or non-functional, though usually functional. The test designer selects valid and invalid input and determines the correct output. There is no knowledge of the test object’s internal structure.Definition from Wikipedia
Black box tests assume that test engineers have no knowledge of the application’s internals—they work purely in terms of inputs and the corresponding expected outputs (whether of data or of error reports for invalid input).
For this kind of testing it is possible to follow the same pattern as
for white box testing, but without providing the test engineer with the
instrumentation database (.csmes file) generated by
CoverageScanner. The application itself will generate an execution
report which can be analyzed after the testing cycle. However, this
approach has the disadvantage that test engineers cannot manage their
own execution reports (e.g., to add comments).
Squish Coco has a facility which makes black box testing possible
without disadvantaging test engineers. The CoverageBrowser tool can
generate an instrumentation database (.csmes file) which does not
contain any source code or source browsing information. This
allows test engineers to manage the list of tests and to view the
statistics, but it does not provide any source level coverage information.
Unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit could be an entire module but is more commonly an individual function or procedure. In object-oriented programming a unit is often an entire interface, such as a class, but could be an individual method.Definition from Wikipedia
CoverageScanner supports the saving of the code coverage data generated by each unit test using its library. (see Chapter 32.2)
As long as all objects are well compiled2, it is possible to merge the unit tests instrumentation database into those of the real application. And so see the code coverage of the unit tests on the main application.
Squish Coco allows us to annotate the execution report with the execution name and state. This does not require any specific library—the information can be inserted directly by editing the execution report before and after the execution of a test item. This makes it possible to integrate the report into an automatic testing framework or for use with a test executing script. (see Chapter 32.1)
This chapter explains the code metrics that are supported by Squish Coco.
The eLoc metric measures the effective number of lines in a piece of code. An effective line is a line which contains statements that produce executable code.
When computing effective lines of code, Squish Coco ignores
This metric is more precise than simply counting the number of lines in a source code project since it skips non-functional code.
The cyclomatic complexity is a code complexity measure proposed by Thomas J. McCabe, Sr. It is a strictly positive number that expresses the complexity of a program. A function with a cyclomatic complexity higher than 10 is usually considered to be difficult to maintain: It contains too many branches, switch/case statements or loops.
In structured programming languages like C++ and C#, the McCabe complexity of a piece of code is
| McCabe = Branchings + Functions . |
In this equation, Branchings is the number of binary branch statements (see below), and Functions is the number of functions in the code.
The binary branching statements in the code are counted in the following way:
if (...) ... else ...” statement counts as one
binary branch, and also a single “if (...) ...”
statement: Both contain a choice between two different execution
paths.
while (...) ... ” or
“for (...) ... ”, also counts a one binary branch
statement, since it contains a condition that offers the choice to
stay in the loop or exit it.
switch (...) { case ...: ... }” construct with
n case labels counts as as n − 1 binary branches
because it can be simulated by n − 1
“if (...) ... else ...” statements.
The McCabe complexity has a several properties which distinguish it from eLOC:
if (...) ... else ...” statement
somewhere in a function increments the metric by one. It does not
matter if the selection statement is nested or at the beginning of
the function and it does not grow exponentially.
In general, for measuring the complexity of a function, the McCabe approach is more useful because the measurement is only dependent of the structure of the code. The code style and comments has no impact on it.
But, as soon as a complete file or a class is analysize, even it is difficult to interpret. The reason is that the McCabe value grows with the amount of functions in a piece of code no matter if the functions in a class are complex or not. In this case, the number of effective lines of code is certainly easier to interpret.
