Part I
Overview
Chapter 1 Squish Coco – A code coverage tool for Tcl, QML, C# and C/C++
Squish Coco is a complete code coverage tool chain for Tcl, QML, C# and C/C++ programs (including SystemC programs). It runs on macOS, Linux™ and Microsoft® Windows. For a full list of supported compilers, see the company website.
Squish Coco analyzes the way an application runs, e.g. as part of a test suite. The results can then be used to make the tests more efficient and complete. In particular, Squish Coco helps to
- find untested code sections.
- find redundant tests which can then be eliminated.
Squish Coco can identify which portions of the source code are covered by a test. It can detect whether a new test covers lines in the source code that the existing tests do not cover.
- find dead code by displaying code that is never executed.
- calculate the optimum test execution order so as to maximize
test coverage for each run.
This is particularly useful for manual testing.
- analyze two separate versions of an application and compare the
differences.
This makes it possible to see which tests are affected by source code modifications and also to get some measure of the test coverage of a patch or hot fix.
- measure the execution time of programs and tests.
Squish Coco can be used at every stage of testing and for all testing methodologies (unit tests, automatic tests, manual white box tests, black box tests, etc.). A test suite can be divided into independent parts: Squish Coco makes it possible to merge multiple execution reports into a single unified report.
The main components of the Squish Coco package are two separate tools and an add-in:
- CoverageScanner – a program to analyze and instrument a Tcl, QML, C# and C/C++ application.
- CoverageBrowser – a program to display and manage the results of the coverage analysis.
- The Visual Studio® Coco Wizard – an optional add-in to configure code coverage for C# or C++ projects that are developed under Microsoft® Visual Studio® and similar IDEs.
There are also several helper programs for report generation, coverage data manipulation and license management.
1.1 CoverageScanner – Instrumentation as part of the build process
CoverageScanner instruments the source code that is seen by the compiler without touching the original code. It generates an instrumented executable, shared library or plugin. The instrumentation keeps track of the code that is executed when the executable is run or when the library or plugin is used. This allows measuring e.g. how many times each line is executed, or whether a line was executed at all. When the instrumented program terminates, it produces an execution report.
In order to get high-quality coverage measurements, CoverageScanner not only instruments at the function and statement level but also records statement coverage and decision coverage information. Another kind of instrumentation supports measurements of code execution times.
Optionally it is also possible to insert the test name and execution status (i.e. “passed” or “failed”) into the report. This can be done by a script or directly from the application. CoverageScanner can then be integrated into a test framework (like CppUnit or CxxTest), where it generates separate coverage information for each test.
CoverageScanner is a command line tool that replaces the compiler normally used to build the executable, library, or plugin. It inserts special instrumentation instructions into the preprocessed source code and then uses the original compiler to compile the modified code. At the same time, a database (the .csmes file) is generated for later analysis. It contains the list of instrumentations and a copy of the instrumented source code.
After compilation, the instrumented executable can be run as usual (and an instrumented library or plugin can be used via the program that accesses them), but now, when the program terminates, an execution report is produced (the .csexe file).
CoverageScanner works with almost any standard compiler, including GNU gcc, GNU g++, Microsoft® Visual Studio® 6.0, .NET, Express, and Embedded C++, Intel® C++, etc. For most other compilers, CoverageScanner can be easily configured.
1.2 CoverageBrowser – Viewing, analyzing and managing code coverage results
The execution report in the .csexe file can be analyzed and managed with the CoverageBrowser tool. This program has a GUI (graphical user interface) by which the user can browse and manage their execution reports. This makes it straightforward to locate untested or unreachable code and inefficient tests.
A summary of CoverageBrowser’s abilities:
- Adding comments to instrumented source code lines.
- Displaying the list of executions in a tree view.
- Marking portions of code which cannot be tested as “Manually Validated”.
- Browsing through the instrumented code.
- Switching between the coverage levels: decision coverage, condition coverage and statement block coverage.
- Displaying a detailed
explanation of the state of the instrumentation. This includes:
- User comments.
- The state of each instrumented statement: executed, not executed, partially executed.
- The number of times an instrumented statement was executed.
- A list of the tests in which an instrumented statement was executed.
- Retrieving the execution status from an automated test suite.
- Code coverage analysis of unit tests.
- Exporting results to spreadsheets.
- A test benefit analysis mode makes it possible to see which additional code coverage a set of tests adds.
- A source code and function browser which displays code coverage statistics for each file, each class, each namespace and for each Tcl, QML, C# and C/C++ function.
- Generation of a report in HTML format. It includes statistics per source file, methods, executions, and a list of unexecuted code lines.
- Black box testing support.
- The ability to compare coverage data for different versions of the software.
