• Overview
• Documentation
  • Code Coverage
  • Quick Start and Tutorial
  • CoverageBrowser
  • CoverageScanner
  • Command Line Tools
  • FAQ
  • Appendix
• Release Note
• License

Part I
Code Coverage Overview

• Code Coverage Overview
  • Code instrumentation
    • Detection
    • Code insertion
    • Result of coverage analysis
  • Performance
  • Statistics
• Testing Methodologies
  • Hit vs Count
  • Strategies
    • Manual tests
    • Black box tests
    • Unit Tests
    • Automatic Tests

Chapter 1  Overview of Squish Coco’s Code Coverage Analysis

CoverageScanner is a C++ program that—in effect—replaces the usual compiler. It preprocesses the source code using the native preprocessor program, in the course of which it inserts instrumentation code, and at the end it compiles and links the project file, just the same as for a normal build.

CoverageScanner’s instrumentation consists of:

1. declaring a static table that contains the of instrumentation results.
2. generating a library which produces an execution report upon program termination.
3. adding the instrumentation code itself for each relevant C and C++ statement or Boolean expression.

The compiler, linker and preprocessor used are from the native environments (gcc, Visual Studio, etc.). These tools are used transparently and the developer has only to prepend ’cs’ to the name of the compiler executable to activate the code coverage ¹. For example, using ’csgcc’ instruments the source code and uses the ’gcc’ compiler to generate objects.
Another code coverage technique involves adding measurement points according to debug information. Since CoverageScanner produces instrumented code during the generation phase, it is possible to analyse the optimized code and achieve a more fine-grained analysis than some other tools achieve. (Many compilers produce unusable debug information on optimized code, but this doesn’t affect CoverageScanner.)
CoverageScanner’s principle code coverage analysis is not to highlight executed source lines, but rather to place marks on execution paths. This makes the analysis independent of the source code’s formatting and groups all sequential instructions together in a single measurement point. This also minimizes the extra memory and processing overhead that the instrumented code incurs.

1.1  Code instrumentation

This section describes the work CoverageScanner does to instrument the code, with each aspect described in its own subsection.

1.1.1  Detection

CoverageScanner parses all C++ language constructs and detects:

1. Execution paths.
2. Boolean expressions which produce different execution paths (if, for, while, switch, …).

To put this in concrete terms we present the following very simple example function:

CoverageScanner first analyses the sequential statements to record each execution path. It determines which instructions need to be instrumented and places a flag to record their execution. To avoid performance problems, CoverageScanner groups all sequential instructions together and records their execution only once: before

coverage instrumentation is usually called Statement Coverage.

In the following version of the example, the instrumented statements are underlined:

In this example, lines 3 and 9 are not monitored since lines 3, 9 and 10 are sequential operations. This doesn’t have an impact concerning coverage because if line 10 is executed then lines 3 and 9 must have be executed before it.

CoverageScanner’s analysis of conditions determines which expressions are used in conditional statements (if, while, for, …). These statements separate the execution path into two or more branches. Code coverage monitors execution by recording if the condition was true or false during the application’s execution.
This kind of code coverage instrumentation is usually called Decision Coverage.
In the example, the instrumented conditions are underlined (the instrumented statements are displayed in bold):

The analysis of Boolean expressions is used to determine which combinations are used in a conditional statements’ (if, while, for, …) Boolean operations (and, or, …). This makes it possible to have a more fine grained analysis of why a particular code segment was executed.
This kind of code coverage instrumentation is usually called Condition Coverage.
For example, the statement return 0; of if (a || b) return 0; can be executed if a or b is true. Analysis of the Boolean expressions will record if variables a and b were true or false. This tells us if the statement ‘return 0;’ was executed, which it will be if either or both of the two expressions was true during the test.

In the example, the instrumented Boolean expressions are underlined:

Squish Coco’s analysis is not limited to control statements (such as do…while or if…then constructs). For example, CoverageScanner also instruments the assignment of Boolean expressions. Constant or static expressions are not instrumented since their values are computed at compile time or once during program initialization.

In the example, the instrumented Boolean expressions are underlined:

The statement static bool a = x && y is not instrumented because it is static. And the statement bool c = b is not instrumented because the expression is a value (albeit a Boolean value), not a Boolean expression.

1.1.2  Code insertion

After the detection phase, CoverageScanner inserts the instrumentation code itself as the following examples illustrate. (Note that the inserted code is displayed in bold blue.)

Sequential Statement

The statements are instrumented before their execution. The instrumentation consists in allocating a Boolean variable which detects if the code was executed or not.
Example: 

   1 a=foo();
   2 a++;
   3 break;

will be changed into:

   1 a=foo();
   2 a++;
   3 { inst[0]=1; break; }

 inst[0]  is set to 1 if the ‘break’ statement is executed.

Conditional Statements and Boolean Expressions (full instrumentation)

For Boolean expressions the same principle applies except that in addition to recording the execution itself, the state (true or false) is also recorded.

Example: 

   1 if ( a<b )
   2   return 1;

will be changed into:

   1 if ( (a<b) ? inst[0]=1 : inst[1]=1,0 )
   2   return 1;

 inst[0]  is set to 1 if the Boolean expression a<b was true.
 inst[1]  is set to 1 if the Boolean expression a<b was false.

Conditional Statements and Boolean Expressions (partial instrumentation)

In some cases, recording the value of a Boolean expression is unnecessary for sequential statement instrumentation. For example, the statement if (b) return 0; else return 1; is completely covered by a statement coverage—after all, recording whether b becomes true or false doesn’t provide any extra information. Similarly, for the statement if (b) return 0; it is only necessary to check if b was false. By default, Squish Coco suppresses the generation of redundant instrumentation in order to minimize the instrumented code’s size and to maximize execution speed.

Example: 

   1 if ( a<b )
   2   return 1;

will be changed into:

   1 if ( a<b )
   2   return 1;
   3 else inst[0]=1 ;

 inst[0]  is set to 1 if the Boolean expression a<b was false. Since the statement coverage records the instruction return 1;, it is not necessary to record if a<b was true;

The generated instrumentation code that needs to be inserted to provide statement coverage for the earlier foo() function example means that the function’s code is transformed to the code shown here:

If we insert the instrumentation code necessary to support decision coverage into this example, the resulting code will look like this:

If we insert the instrumentation code necessary to support condition coverage into this example, the resulting code will look like this (except that we have wrapped one line to fit better on the page):

Here is what the code would look like if we inserted the partial instrumentation code for condition coverage (again, with one line wrapped):

1.1.3  Result of coverage analysis

The CoverageBrowser program features a graphical user interface which it uses to display the analysed results of the instrumentation. CoverageBrowser uses color coding to indicate statement status. For this manual we have used the same colors as the program, but in addition we have applied text effects to aid those reading a monochrome print out.

• bold green indicates statements which are executed.
• shaded bold red indicates statements which are not executed.
• italic orange indicates statements which are partially executed. These are Boolean expressions or conditional statements which were always true or always false.

The output for the example would be:

The expression i<100 is partially executed because is was always true. The expressions i==50 and i==30 are always false, yet they are shown as being fully executed. Since these expression are incorporated in an if ... then statement, it is not necessary to check if they were true (here the non-execution of the break; and found=true; statements provide the same information).

1.2  Performance

The insertion of instrumentation increases the code size and also affects the instrumented application’s performance—it will use more memory and will run slower.
For non-conditional expressions the instrumentation code involves a simple write to a fixed memory location. However, for conditional expressions more detailed analysis is needed and this is more computationally expensive.
Overall, an instrumented application will be from 60% to 90% larger, and will run from 10% to 30% slower.

1.3  Statistics

Some developers write more lines of code than others simply by using a particular coding style—for example, putting opening braces on a line of their own rather than on the same line as, say, an if statement. In view of this, Squish Coco computes its statistics using a more subtle measure that isn’t susceptible the the vagaries of coding style: its calculations are based on the number of executed instrumented instructions compared with the total number of instrumented instructions.

Each simple instrumented instruction (return, break, last instruction of a function, etc.) is recorded by one single instrumentation. A fully instrumented condition in an IF...THEN...ENDIF statement uses two instrumentations: one for the true case and one for the false case. If the code is only partially instrumented, only one condition is recorded (either the false case or the true case).

The statistics themselves depend of the type of instrumentation. It is not generally possible to compare code coverage at branch level with that at decision level: having 80% coverage at decision level, does not tell us anything about the coverage at branch level, which could be bigger or smaller. The only thing we can be sure of is that reaching 100% coverage is more difficult with condition coverage than with decision coverage and than branch coverage.
In our example, the coverage is between 60% and 75%. The following table shows the details for each instrumentation.

In the examples below, the details of the calculations are displayed using subscripts. The first number is how many instrumentations were executed and the second is the number of instrumentations in total.

Chapter 2  Testing Methodologies

This chapter discusses the various testing strategies that can be carried out so as to make the best use of the Squish Coco tools.

2.1  Coverage Hits vs. Counts

If we want to set very stringent standards for an application to ensure that it is of the highest possible quality, executing just portions of its code, or executing code just once, is insufficient.

To address this issue the Squish Coco tools provide two possible approaches:

Code Coverage Count

We can require that each portion of instrumented code is executed a minimum number of times. This can be achieved by adding to the instrumented code so that it counts each execution.¹ Naturally, setting such minimums, and the extra bookkeeping itself, can make the application consume more memory and run a bit slower. An important disadvantage of this approach is that loops in the source code end up having a high count with only a single execution, so counting this way isn’t always useful.

Test Coverage Count

We can require that each portion of instrumented code is executed by a minimum number of test items. This avoids the code coverage count disadvantage regarding loops, since by setting a minimum number of test executions for each portion of instrumented code, we give equal value to loops, recursions, and to instructions that are executed only once.

2.2  Testing Strategies

Squish Coco can be adapted to fit in with many different testing strategies, from unit tests during early development all the way through to complete product validation (e.g., acceptance tests). CoverageBrowser supports the merging of code coverage results at each stage of testing, and so provides a precise overview of the software’s overall quality.

2.2.1  Manual White Box Tests

Interactive white box tests are easy to do using Squish Coco: once the application has been compiled with CoverageScanner, the test engineer can execute the application, and after each execution they can import the execution report into CoverageBrowserfor analysis.

2.2.2  Manual Black Box Tests

Black box tests assume that test engineers have no knowledge of the application’s internals—they work purely in terms of inputs and the corresponding expected outputs (whether of data or of error reports for invalid input).

For this kind of testing it is possible to follow the same pattern as for white box testing, but without providing the test engineer with the instrumentation database (.csmes file) generated by CoverageScanner. The application itself will generate an execution report which can be analyzed after the testing cycle. However, this approach has the disadvantage that test engineers cannot manage their own execution reports (e.g., to add comments).

Squish Coco has a facility which makes black box testing possible without disadvantaging test engineers. The CoverageBrowser tool can generate an instrumentation database (.csmes file) which does not contain any source code or source browsing information. This at least allows test engineers to manage the list of tests and view the statistics, but does not provide any source level coverage information.

2.2.3  Unit Tests

CoverageScanner supports the saving of the code coverage data generated by each unit test using its library. (see chap. unit-test)

As long as all objects are well compiled², it is possible to merge the unit tests instrumentation database into those of the real application. And so see the code coverage of the unit tests on the main application.

2.2.4  Automatic Tests

Squish Coco allows us to annotate the execution report with the execution name and state. This does not require any specific library—the information can be inserted directly by editing the execution report before and after the execution of a test item. This makes it possible to integrate the report into an automatic testing framework or for use with a test executing script. (see chap. execution-name-status)

1

This can be done using CoverageScanner’s --cs-on command line option.

2

Well compiled means that a source file is not be compiled twice or at least it is not be compiled with different preprocessor options.