Squish Coco
Code Coverage Measurement for C/C++
Part III |
| Execution Name | * |
TESTS using the wildcard syntax:| Execution Name | TESTS/* |
TESTS using the regular expression syntax:| Execution Name | =TESTS/.* |
Here some examples of renaming execution:
-
To put all executions in the directory
TESTSset:
Actual Execution Name =.*New Execution Name TESTS/& - To move all executions in the directory
TESTSin the directoryOLDset:
Actual Execution Name =TESTS/(.*)New Execution Name OLD/\1 - To rename all executions in all directories in
testname [directory]set:
Actual Execution Name =([^/]*)/([^/]*)New Execution Name \2 [\1]
7.2 Loading an Execution Report
The execution report is produced upon application exit.
Its name is defined by the initialization function
__coveragescanner_install of the CoverageScanner library
(see chap. coveragescanner_install).
Its extension is followed by ".csexe".
It contains the list of all executed code segments for each application run.
The execution report is never overwritten but
execution contents are appended.
To load an execution report click on "File->Load Execution Report"
or the icon 
on the toolbar.
The dialog as shown in fig:coveragebrowser_loadexec_fig will appear.
The file can be loaded directly or using a script.
To load directly, click on "file",
enter the path of the ".csexe" file to load
to the free form input box or use the browse button.
If the option "Delete after transferring" is selected, the execution report file
will be deleted after transfer.
The option "When file becomes modified" permits to reopen this dialog
automatically when the execution report is modified.
The "Name" field allows the user to specify the name of the imported instrumentation
if this is not specified in the execution report (see chap. executionreportname). Also, it is
possible to set the execution status (passed, failed or to be checked manually) of all imported executions.
If more than one instrumentation is imported an index is appended to this default name.
The option "Import Preprocessing" permits to select the behaviour
in the case of conflicts or redundant executions:
- "Ignore Duplicate Executions"
- Executions which have executed the same code than an execution which are already imported are ignored.
- "Import Duplicate Executions"
- Executions are imported if at least one instrumented source code line is executed.
- "Import Duplicate And Empty Executions"
- All executions are imported.
- "Merge all executions with the same name together"
- All executions with the same name are merged (i.e. the execution count of each instrumentation is added).
Invalid executions are not imported and
a summary shows when the operation is completed (see figure fig:coveragebrowser_loadreport)
when "Display import summary" option is selected.
If the execution report is not accessible through the file system,
a script can be used. The script has only to print the contain of
the execution report to the standard output (stdout). The standard error
output (stderr) is displayed on the screen and can be used for debugging
purpose. On success, the script must exit with the value 0.
7.3 Execution Comparison Analysis Mode
The execution comparison analysis mode if activated by clicking on the button 
.
In this mode, CoverageBrowser does not display the coverage of a set of tests but the lines
covered by an execution which are not covered by a set of reference executions.
In other words, CoverageBrowser shows what an execution is covering more than a set of other executions.
The list of reference executions are the executions checked in the execution list.
Double-clicking on the name of the execution permits to select the test to analyse.
Only the instrumented lines which are executed by this execution are shown, the other are in the state "hidden".
Also the coverage statistics displayed in the source list are only containing the percentage of
instrumented statements which are only executed by this the selected execution.
Chapter 8 Source Browser Window
The "Sources" window can be displayed by clicking
on "View->Sources".
Each item is a C and C++ source file and the sub-entries are the list
of included headers which have been instrumented. When an item
the "Source Window" will be displayed.
The "Sources" window displays rudimentary code coverage statistics
for each source code file. The color of each item is selected according to code coverage
statistics for each file and the watermarks.(see chap. watermarks)
The input field 
permits to filter the output using regular expressions
(see chap. sec:coveragebrowser-filter).
The item text which can be filtered is the full path of the source file. (example: c:\directory\file.cpp)
Chapter 9 Method Browser Window
The "Methods" window can be opened by clicking
on "View->Methods".
The "Methods" window displays a the code coverage
statistics of each C and C++ functions, classes and namespaces.
Clicking on a function permits to show all instrumented lines of it
in the "Source Viewer".
The input field permits to filter the output using regular expressions
(see chap. sec:coveragebrowser-filter).
The item text which can be filtered is the symbol name including the class name and the namespace.
(example: MyNamespace::MyClass::MyProc)
Chapter 10 Source Viewer Window
10.1 Source Display
The "Source Viewer" window can be displayed by clicking on "View->New Source Window".
The "Source Viewer" window displays the source file or its C or C++ preprocessed view.
Clicking on 
enables the user to toggle between the 2 different views.
The source code is colored with code coverage instrumentations.
The colors used are described in section sec:color_convension.
By selecting an area with the mouse, corresponding instrumentations are highlighted and a
detailed description of them is displayed in the "Explanation"
window (see chap. code-coverage-explanation). It is possible to navigate between
instrumentations using the navigation buttons 
and 
.
Navigation buttons in yellow, blue, red and green permit to jump to the next or previous
comments, manually validated instrumentations, non-executed code parts or executed code parts.
Clicking on the source code selects the nearest instrumentation.
If a comment is entered for an instrumentation, the icon 
is displayed in the margin.
On the right side, CoverageBrowser displays the test coverage
count1
or the code coverage count2
for each line.
If a source code line contains more than one instrumentation,
CoverageBrowser display the range of their counts.
10.2 Color Convention
Instrumentations are displayed in a source window using different colors:
- Green - "Executed"
- An instrumentation is displayed in green when the code has been executed.
- Orange - "Partially Executed"
- An instrumentation is marked as "Partially Executed" when it is not completely executed. This occurs when a Boolean expression was only true or false for example. In the case of a source code line which contains more than one instrumentation, the line is marked as "Partially Executed" when one of its instrumentations has not been "Executed". A detailed information is displayed in the "Explanation" window (see chap. code-coverage-explanation).
- Red - "Never Executed" or "Execution count too low"
- An instrumentation is displayed in red when the code is never executed or when the execution count is lower that than the execution count requested.
- Magenta - "Dead-Code"
- An instrumentation is displayed in magenta when the code cannot be executed.
- Blue - "Manually Set To Be Executed"
- The user has the possibility to mark an instrumentation as ’Manually Validated’. This is usually to exclude dead code or code which cannot be tested for code coverage statistics. This state is only relevant if executions are in a "Never Executed" or "Partially Executed" state.
- Gray - "Unknown" or "Hidden"
- Gray is used when no information about instrumentation is available. This occurs when no executions are selected or when comparing executions of tests (see chap. test-benefit).
10.3 Comments
10.3.1 Editing Comments
It is possible to add a comment by selecting an instrumentation and clicking on
the context menu entry "Add/Edit Comment", the main menu entry
"instrumentation->Add/Edit Comment" or the icon on the toolbar.
The "Comment" Window fig:coveragebrowser_comment_edt appears and allows a comment to be edited. The most recently entered comments can be retrieved by clicking on the "Last Comments" selection field. Basic text formatting is possible using the integrated toolbar buttons (see comments_shortcut).
The comment is printed in the explanation in a yellow box and the icon ()
is displayed in the source window near the line number.
10.3.2 Removing Comments
It is possible to remove a comment by selecting an instrumentation
and clicking on the context menu entry "Remove Comment", the main menu
entry "instrumentation->Remove Comment" or the icon 
on the toolbar.
Chapter 11 Code Coverage Explanation Window
The "Explanation" Window fig:coveragebrowser_expl_fig
is a docking window which is automatically updated with a detailed
description of the selected instrumentations of the source window.
For each instrumentation, the following information is displayed:
- A short description of the instrumentation state (see chap. sec:color_convension).
- The preprocessed source code which is concerned by the instrumentation.
- For Boolean expressions, the truth-table which shows executed and unexecuted states.
- The list of executions which are executing the portion of code.
- User comments.
CoverageBrowser displays the truth-table in the case of a Boolean expression which is partially executed.
The truth-table indicates which value the expression has or has not reached during execution.
Example: the truth-table truth_table indicates that the expression was false but not true.
Chapter 12 Statistic Window
The "Statistic" Window fig:coveragebrowser_statistic
is a docking window which is automatically updated with the
code coverage statistic for the whole project.
If the coverage level is greater than one, the "Statistic" Window displays
the statistics of the current level and the level one.
Chapter 13 Filter using wildcard expression or regular expressions
CoverageBrowser provides a generic filtering mechanism of rows using wildcard or regular expressions.
Wildcard expressions are activated per default and regular expressions are selected when the expression starts with
an equal sign (’=’).
Clicking on the filter icon converts the expression from wildcard into regular form as far as this is possible
and vice versa.
13.1 Wildcard Expression
|
Example: foo*bar match any tests containing the string foo followed by bar.
13.2 Regular Expression
The first character must be ’=’ to activate the regular expressions.
13.2.1 Pattern matching
|
13.2.2 String substitution
|
Chapter 14 Code/Test Coverage Level
The menu entry "Instrumentation->Level:x" permits to set the targeted code coverage count or, if the compiled with instrumentation hit support1, the targeted test coverage count.
The level is corresponding of the number of code/test coverage count necessary
to consider that an instrumented code is executed.
Example: Setting the level to 10, will made necessary to execute 10 times
the each line of the source code if compiled with code coverage count.
If compiled with code coverage hit, 10 execution runs need to execute
each lines of the source code.
The menu entry "Tools->Test Coverage Count Mode" and the button 
permits to switch between code coverage count and test coverage count analysis. This simulates the behaviour of the compilation with code coverage hit support2 when the project is compiled with code coverage count support3.
Chapter 15 Code Coverage Algorithm
CoverageBrowser displays the code coverage analysis (branch, decision or condition) generated be CoverageScanner.
But "Instrumentation->Coverage Method->Branch only" permits to reduce the analysis to the code coverage of branches.
This produces the same result as compiling with the --cs-branch of CoverageScanner.
"Instrumentation->Coverage Method->Decision, Condition and Branches" permits to show the code coverage analysis at the level defined at the compilation.
Here a short overview of the command line options necessary for each code coverage analysis method:
|
Chapter 16 Optimized Execution Order
CoverageBrowser is able to calculate an optimized order of the executions
(i.e.: the order of tests which permits to maximize the code coverage after
each execution).
This order is specially adapted to manual testing: following this order permits
to execute first the tests which are giving a high code coverage and so detecting rapidly
errors in the first test executions.
To calculate the execution order proceed as follows:
- Select a set of executions in the "Executions" window.
- Click on "Tools->Optimized Execution Order…". The window fig:optimized_execution_order_fig will de display.
Chapter 17 Comparing Code Coverage of Two Software Releases
CoverageBrowser is able to instrumentation database together in order to:
- check is the modified/unmodified code is correctly tested.
- find which tests are impacted by a source code modification.
This feature is particularly adapted to compare two releases together which contains small modifications (bug fixes only) and to limit the tests of the modified code only.
In this mode CoverageBrowser uses the following typographic rules:
|
CoverageBrowser comparison and difference algorithm is particularly designed for C and C++ source code and ignore white spaces and modifications in comments.
17.1 Reference Database
The reference database is the base instrumentation database which are used for the comparison.
To select it click on "Tools->Compare with…" and select a .csmes database.
Switching the working database with the reference database can be performed by clicking on "Tools->Switch databases".
Once the reference file loaded, additional filter possibilities are available in the "Executions", the "Sources" and the "Methods" window. This filters permits to show/hide, modified, new, deleted or identical procedures and source files.
The "Executions" window displays a mix between the executions of the reference and the current release:
- stroked executions are only available in the reference release. The statistic computed is corresponding to the coverage for the reference release. This value can be interpreted as an expected code coverage when this tests get re-executed on the current release.
- underlined executions are new tests.
- executions available in both version are not highlighted.
17.2 Coverage analysis of modified/unmodified source code
CoverageBrowser is able to limit the code coverage analysis to the modified (resp. unmodified) functions.
When selecting the coverage analysis on the modified (resp. unmodified) functions only,
CoverageBrowser treat all unmodified (resp. modified) functions as if they are not instrumented.
Limiting the code coverage analysis to modified functions can be a practical
way to verify that the new features are tested and to identify the list of tests
which have are impacted by a modification.
To limit the code coverage to modified function (resp. unmodified functions) click on "Tools->Analysis of Modified Methods" (resp. "Tools->Analysis on Identical Methods").
Chapter 18 Instrumentation Database
18.1 Merging Instrumentations
Clicking on the menu entry "File->Merge with…"
permits to import the executions, the source code and the instrumentations from other .csmes files.
Comments and code mark as validated are merged together.
18.2 Importing Unit Tests
Clicking on the menu entry "File->Import Unit Tests…" permits to import the execution report of unit tests into the current application. Only execution reports of source file present into the main application are imported, instrumentation of other source files (for example test code) are ignored.
Chapter 19 Exporting Statistics
19.1 EMMA-XML Report
Clicking on the menu entry "Reports->Export Statistics in EMMA-XML Format…" permits to export code coverage statistics of in EMMA-XML format. This output format generate only global statistics in a format that is compatible with EMMA. This format allows to use Squish Coco in tools that provide support for EMMA, notably giving an easy way to use Squish Coco with continuous integration servers like Hudson.
EMMA defines only four categories for coverage: classes, methods, blocks, and
lines. Due to the fact that blocks does not make sense with
Squish Coco,
this cathegories have the following meaning:
| EMMA category | Meaning |
| classes | A class is considered executed if one of its method is called. Code
which are not in a class are located in the class "" (empty). |
| methods | A method is covered if it was called. |
| blocks | Code coverage at branch level. |
| lines | Line code coverage. (if compiled with line coverage support) |
| conditions | Decision/Condition line coverage. (if compiled with Decision/Condition coverage) |
19.2 JUnit Report
Clicking on the menu entry "Reports->Export JUnit Report…" permits to export the test result as JUnit report. This report does not contain coverage data and only list the test execution result (passed or failed) for each test item.
19.3 Statistics per Source File
Clicking on the menu entry "Reports->Export Statistics per Source File" permits to export code coverage statistics of each source file in CSV format.
The file contains the coverage of all combination of source files and executions selected and the execution status (see figure fig:statistic_csv_fig). Comma and semi-colon can be selected as separator in the CSV file using the "File Type" input field.
19.4 Statistics per Method
Clicking on the menu entry "Reports->Export Statistics per Method" permits to export code coverage statistics of each function and procedure in CSV format.
19.5 HTML/XML Report
Clicking on the menu entry "Reports->Generate Report…" permits to export code coverage statistics (per methods, source files, executions, …) of the selected executions. It permits also to list the manually validated and unexecuted code parts.
19.6 Text Report
Clicking on the menu entry "Reports->Generate Text Report…" permits to generate a small text report in the form of one line per executed/unexecuted item. A distinct line format can be specified for executed or unexecuted lines.
Following keywords are recognized:
- %f
- Source code file name
- %l
- Line number
- %c
- Column number
- %m
- Explanation
Example: setting the field "Unexecuted Code Fragments" to "%f:%l: %m" will
create a text file which contains all unexecuted code parts. Each line will look like as follows:
foo.cpp:55: Unexecuted: 'return;'
Chapter 20 Preferences Dialog
20.1 Save/Load Project
- "Save/Restore window position"
- If this option is selected, the position of all windows and toolbars will be restored upon application restart.
- "Reload automatically the last project"
- If this option is selected, the last project opened will automatically be reloaded upon application restart.
- "Saves project automatically on exit"
- Saves the project file automatically without asking on application exit.
20.2 Comments
- "Minimum Comment Size"
- The minimum comment size, is the minimum length requested for a comment.
- "Do not request a comment when setting an item to the ’manually validated’ state"
-
This option is used to allow the user to manually modify
the state of an instrumentation without entering a comment.
Enabling this option should be avoided because modifying the state of an instrumentation should be performed with a valid reason which should be recorded as a comment.
20.3 Watermarks
Watermarks are trigger values that control the background color of:
- the instrumented source files in in the "Sources" window.
- the instrumented classes or namespaces in the "Methods" window.
- the instrumented functions, methods or procedures in the "Methods" window.
Description:
- "Medium/High Coverage Level"
- If the statistic is above this value, the background color is set to green. Otherwise, the color is orange.
- "Low/Medium Coverage Level"
- If the statistic is below this value, the background color is set to red. Otherwise, the color is orange.
20.4 Cache
Description:
- "Execution"
- Maximum number of executions loaded into the RAM.
- "Source"
- Maximum number of source files loaded into the RAM.
Chapter 21 Miscellaneous
21.1 Command Line Arguments
Syntax: coveragebrowser -m <csmes_file> -e <csexe_file> ...
Where:
- -m <argument> | --csmes=<argument>
- Load an instrumentation database.
- -e <argument> | --csexe=<argument>
- Open the import dialog which permits to load a specified execution report.
This command line argument requires the command line option --csmes.
| ©2011 froglogic GmbH |  squish@froglogic.com |