This chapter describes the installation and setup of Squish Coco on your machine and a later update of software and licenses.
There are two kinds of licenses, node-locked and floating licenses.
A node-locked license is bound to a specific machine and a specific user account. On that account, you can run as many Squish Coco processes as you like. Note that if you run Squish Coco from a Continuous Integration Server, the server has a separate account; in this case Squish Coco needs an additional node-locked license.
Floating licenses are bound to processes, not to computers. One needs as many floating licenses as there are Squish Coco processes running at the same time. A Squish Coco process can be
Every compiler call that is instrumented by Squish Coco is counted separately because it requires one instance of the CoverageScanner to run. Builds with several compilations in parallel should however cause no real problem since a process that cannot get a license will wait until one is freed.
Floating licenses are managed by a license server (see Chapter 28). The server must be running on a machine that is reachable from the machine on which Squish Coco runs. froglogic will provide a license file which determines the maximal number of Squish Coco processes that can be active at the same time. (The license server itself is not counted among the processes that need a license.)
An installer for Squish Coco can be downloaded via https://www.froglogic.com/squish/download/.
The installers for the various platforms supported by Squish Coco have a
common name scheme, but the way they must be executed differs. The
naming scheme is
SquishCocoSetup_
x.y.z_⟨platform⟩.⟨suffix⟩,
where x.y.z is the program version, ⟨platform⟩ describes
the operating system and other details of the installation, and
⟨suffix⟩ is an operating system specific suffix. (In the
noncommercial version, SquishCoco is replaced by
SquishCocoNoncommercial.)
SquishCocoSetup_
x.y.z_Windows_x86.exe
or
SquishCocoSetup_
x.y.z_Windows_x64.exe
and
must be executed.SquishCocoSetup_
x.y.z_⟨platform⟩.run
and must be executed with bash, as in
Note that the CentOS version of the installer is also valid for RedHat Linux™.
SquishCocoSetup_
x.y.z_⟨platform⟩.pkg
.
Click on it and the installer will be started.
If no valid license is present for your account, the installer will run the License Wizard (see Chapter 27) at the end of the installation. The License Wizard will then allow to configure the license.
If you have chosen a floating license, you need to specify a machine on your local network on which the license server program runs.
The license server needs a configuration file to run. It specifies the number of licenses served, the machine on which it runs, and the port used by the server. To specify the configuration file, Squish Coco must be installed first.
Install therefore Squish Coco on the license server machine. After the
installation, cocolicwizard will be started. Ignore it. Instead, run
the command cocolicserver --server-identifier
and redirect
its output to a file. How this is done depends slightly on the
operating system.
This will generate a file machine.txt
in the directory in
which the command was executed. Send it to froglogic, together with
the number of requested licenses (and, if necessary, the port number).
froglogic sends you a configuration file, which is usually called
cocoserver.cfg
. With it, the license server can run. In the
simplest case, write
The server then provides licenses to other machines in the same network. For additional options of cocolicserver, see Chapter 28.
A new license can be installed with with cocolic or cocolicwizard.
To update to a newer version of Squish Coco, download and install it. It will then overwrite the previous version. The license will not be touched, you can continue to use it.
If a new compiler is installed, then the Squish Coco installer should be run again, so that Squish Coco becomes aware of the new compiler. This includes new versions of Microsoft® Visual Studio®: If Visual Studio 2013 was present during the Squish Coco installation and Visual Studio 2015 is added, the Squish Coco installer needs to be run.
If instead the compiler is updated, Squish Coco should work mostly without changes.
In this section we will show a small project with unit tests and show how it can be instrumented. The project is a simple expression parser, and it has not many requirements besides a C++ compiler.
The project replicates in miniature an existing project that has been extend with unit tests and for which we will now use Squish Coco to find out how good the test coverage is. Instrumentation should therefore be non-intrusive and should not change the project very much.
The parser example can be found in Squish Coco’s installation directory.
Under UNIX®, this is the directory /opt/SquishCoco/
or, if
you have installed Squish Coco locally, the subdirectory
SquishCoco/
of your home directory.1 Under macOS, the installation
directory is /Applications/SquishCoco/
.
We will refer to it as the SquishCoco/
directory, wherever
it is located. The Squish Coco examples, together with their supporting
programs, are in SquishCoco/samples/
, and the parser is in
SquishCoco/samples/parser/
. This directory contains three
versions of the program, in the directories parser_v1/
to
parser_v3/
. They represent the parser in different stages of
its development.
The example uses
CppUnit
as its unit test framework. There is a version of CppUnit in the
SquishCoco/samples/
directory, and the parser example is
prepared to use it.
Copy therefore now the content of the whole samples/
directory to your workspace and make parser_v1/
your working
directory. If Squish Coco is installed in /opt/SquishCoco/
, this
is done in the following way:
Make also sure that the Squish Coco tools are in your search path. If they are not so already, you can now do it by writing
(Do not forget the dot at the beginning!) Now programs like coveragebrowser can be called from the command line.
We will use samples/parser/parser_v1/
as our working
directory. It contains C++ source files and header files,
together with an unit test file, unittests.cpp
. The makefile
has been prepared for unit tests, but not for instrumentation. The
instrumentation is done with the help of the bash script
instrumented
. (There are also some files that are needed in
the Microsoft® Windows version. We will ignore them here.)
Run “make
” to compile the program. It is a simple expression
parser and calculator.
We have added some unit tests for the main class, Parser
.
Look into the file unittests.cpp
to see the tests that have
been included. Execute it with make tests
. You will see
that 14 tests have been executed, and also that one of them has
failed. This is for greater realism and also allows us to see later
how Squish Coco handles test failures.
We have kept the instrumentation separate from the main project. The
core of the instrumentation is a short shell script,
instrumented
. It is a simple wrapper, and calling
“instrumented ⟨command⟩” executes ⟨command⟩ with
a few environment variables set. We will do this now. Enter
The first command removes all object files, since we need everything to be recompiled. The second command then compiles the program with instrumentation and runs the tests. That’s all!
We now have a look at what the script has done and how it has done it.
List the contents of your parser
directory:
You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the information that is needed for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .o.csmes are temporary files and are only used during compilation.
This time, the only program that was actually executed was
unittests
, and therefore the only .csexe file is
unittests.csexe
. To see the coverage results, you can
therefore start the CoverageBrowser with the command
Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button to load the data. (This button will also appear als "Import & Delete", depending on the settings of the browser.)
unittests.csexe
file.For the use of CoverageBrowser, see Part VII. We will now rather describe how the instrumentation is done.
The file instrumented
is a short bash script:
At its beginning, the shell script getcoco.sh
sets the shell
variable COCO_WRAPPER_DIR
. It contains the name of the
directory in which Squish Coco is installed. Then there are four
export
statements, and the final cryptic statement executes
the command line parameters of instrumented
. So if you call
./instrumented make tests
, the command
make tests
is executed by the script, but in a
different environment than normally.
The important part of it are therefore the four export
statements. In the first one, the search path is manipulated so that
the programs in /opt/SquishCoco/wrapper/bin/
are searched
first. This directory contains a lot of files with the same names as
the compilers2 that are supported by Squish Coco:
These programs are the compiler wrappers. With the new PATH
,
they are executed instead of the real compilers. The compiler wrappers
are actually symbolic links to a single program,
coveragescanner (see Part IV).
When executed to compile a source file, they create an instrumented
version of the source and then run the original compiler to compile
it.
In the other three
export
statements, additional flags for the compiler
wrappers (see Chapter 9) are set. The most important
option is the first one, --cs-on. If it is not present, the
compiler wrappers are inactive and just call the compilers they
represent. The following options enable MC/DC (see Chapter 5.2.5)
and Multiple Condition coverage (see Chapter 5.2.4).
They are disabled by default because they are sometimes expensive to
execute. Here they are enabled so that you can see all coverage modes
later in the CoverageBrowser.
The resulting script should work without changes for many simple projects. If more customization is needed, it can often be achieved by adding more options to COVERAGESCANNER_ARGS.
It is also convenient to add make targets to handle the
files generated by CoverageScanner. In the parser
directory,
the Makefile
has been changed in the following way:
Since the .o.csmes
files are needed only for compilation,
they can be deleted whenever the .o
files are deleted (which
is that what make clean
does). The .csmes
and
.csexe
files are more precious and should only be deleted
when all generated files are removed. Therefore we have added their
deletion statements to the distclean
target.
The parser example can be found in the directory ⟨Windows Coco⟩\parser
.
This directory contains three versions of the program, in the
subdirectories parser_v1
to parser_v3
. They
represent the parser in different stages of its development.
The example uses
CppUnit
as its unit test framework. There is a version of CppUnit in the
⟨Windows Coco⟩\squishcoco
directory, and the parser example is prepared
to use it.
Since these directories are write-protected, you need to create your
own working copies. Copy therefore the two directories
⟨Windows Coco⟩\squishcoco\parser
and
⟨Windows Coco⟩\squishcoco\cppunit-1.12.1
to a directory of
your choice. Then remove the write protection of the directories and
all the files contained in them.
We will use parser\parser_v1
as our working directory. It
contains C++ source files and header files, together with an
unit test file, unittests.cpp
. The Makefile
has
been prepared for unit tests, but not for instrumentation. The
instrumentation is done with the help of the batch file
instrumented.bat
. (There are also some files that are needed
in the UNIX®version. We will ignore them here.)
We will do the compilation of the example on the command line. To get
a command window, execute the batch file CocoCmd.bat
that is
located in the parser_v1
directory. In this window, the
Microsoft® Visual Studio® command line tools (like nmake
) are
accessible, and also the main Squish Coco programs (like CoverageBrowser).
Run “nmake
” to compile the program. It is a simple expression
parser and calculator.
We have added some unit tests for the main class, Parser
.
Look into the file unittests.cpp
to see the tests that have
been included. Execute it with nmake tests
. You will see
that 14 tests have been executed, and also that one of them has
failed. This is for greater realism and also allows us to see later
how Squish Coco handles test failures.
We have kept the instrumentation separate from the main project. The
core of the instrumentation is a short shell script,
instrumented.bat
. It is a simple wrapper, and calling
“instrumented.bat ⟨command⟩” executes ⟨command⟩
with a few environment variables set. We will do this now. Enter
The first command removes all object files, since we need everything to be recompiled. The second command then compiles the program with instrumentation and runs the tests. That’s all!
We now have a look at what the script has done and how it has done it.
List the contents of your parser
directory:
You see two kinds of files that do not appear as result of normal compilation. The .csmes files contain the information that is needed for coverage measurement, and the .csexe files contain the results of code execution. The files that end in .obj.csmes are temporary files and are only used during compilation.
This time, the only program that was actually executed was
unittests
, and therefore the only .csexe file is
unittests.exe.csexe
. To see the coverage results, you can
therefore start the CoverageBrowser with the command
Then the CoverageBrowser will start with a modal window, "Load Execution Result". Click on the "Import" button to load the data. (This button will also appear als "Import & Delete", depending on the settings of the browser.)
unittests.csexe
file.For the use of CoverageBrowser, see Part VII. We will now rather describe how the instrumentation is done.
The file instrumented.bat
is a short batch file:
The variable SQUISHCOCO
contains the name of the directory
in where Squish Coco is installed. It is set by Squish Coco during installation.
At the beginning, the setlocal
command ensures that the
following commands change the environment variables only temporarily.
Then there are two set
statements, and the final
call
statement executes the command line parameters of
instrumented
. So if you call
“instrumented nmake tests
”, the command
“nmake tests
” is executed by the batch file, but in a
different environment than normally. At the end, endlocal
undoes the changes in the environment variables.
The important part of the script are therefore the two set
statements. In the first one, the search part is manipulated so that
the programs in ⟨Windows Coco⟩\squishcoco\visualstudio
are searched
first.3 This directory contains files with the same names as
the compilers and the linker
The .exe
files in this directory are the compiler wrappers.
With the new PATH
, they are executed instead of the real
compiler. The compiler wrappers are actually copies of a single
program, coveragescanner.exe
(see Part IV). When executed to compile a
source file, they create an instrumented version of the source and
then run the original compiler to compile it.
In the second set
statement, additional flags for the
compiler wrappers (see Chapter 9) are set. The most
important option is the first one, --cs-on. If it is not
present, the compiler wrappers are inactive and just call the
compilers they represent. The following options enable MC/DC
(see Chapter 5.2.5) amd Multiple Condition coverage
(see Chapter 5.2.4). They are disabled by default
because they are sometimes expensive to execute. Here they are enabled
so that you can see all coverage modes later in the CoverageBrowser.
The resulting script should work without changes for many simple projects. If more customization is needed, it can often be achieved by adding more options to COVERAGESCANNER_ARGS.
It is also convenient to add make targets to handle the
files generated by CoverageScanner. In the parser_v1
directory, the Makefile has been changed in the following way:
Since the .obj.csmes
files are needed only for compilation,
they can be deleted whenever the .obj
files are deleted
(which is that what make clean
does). The .csmes
and .csexe
files are more precious and should only be
deleted when all generated files are removed. Therefore we have added
their deletion statements to the distclean
target.
In the following sections we will show additional abilities of Squish Coco. They will require small changes in the code of the project.
The coverage information generated so far has a problem: It covers too many files. The problematic files are those that belong to the testing framework and not to the tested program. Including them would create artificially low coverage rates.
With Squish Coco, one can exclude files from coverage by additional command
line options. In parser_v2
, this has been done. Look into
parser_v2/instrumented
(or
parser_v2\instrumented.bat
under Microsoft® Windows). In it,
three additional command line options have been set, which we will now
explain:
You can use slashes or backslashes with this option—Squish Coco normalizes them internally.
unittests.cpp
and CppUnitListener.cpp
. (The
second file is described below.)
For the next modification, we want to change the project such that we know not only whether a line of code is covered by tests, but also by which tests it is covered. For this we will add calls of the CoverageScanner C/C++ library (see Chapter 10.1) to the code, to tell Squish Coco the names of the tests and where they begin and end.
An updated version of the project can be found in the directory
parser_v2
. The greatest difference to the version in
parser_v1
is that the file CppUnitListener.cpp
has
been added. It is copied almost verbatim from Chapter 38.1.
The file contains a class CppUnitListener
and a new
main() function. The main() function in
unittests.cpp
has been removed, but the file is otherwise
unchanged.
CppUnitListener.cpp
provides a unit test listener which
allows to hook into the framework before and after the execution of
each test. One can thus record additional test information, like the
name and the result of a test, in the code coverage data without
modifying the test code itself. (For a listing of
CppUnitListener.cpp
and an explanation how it works, see
Chapter 38.1)
Now you can execute this program with the same way as its previous
version. View the results in the CoverageBrowser. In the
"Executions" subwindow you can now see the code coverage for
each single test. You can especially see that it was the test
testInvalidNumber
that had failed.
Hover with your mouse over one of the lines in the source window, and select a line that was marked in dark green. This means that it was executed by the tests4. You will see a tooltip with a list of the tests that executed this line. To keep the tooltip short, there is a maximal number of tests to display there. To see a full list, click on the line. Then the full information about the line is shown in the "Explanation" window.
Now consider the following scenario: In a large project, a last-minute patch has to be evaluated. It is not enough time to run the full test suite, but some risk assessment needs to be done. For situations like this, Squish Coco provides the feature of patch analysis. With it, one can specifically display the code coverage for the changed lines of code, and find the tests in a large suite that cover them. One can now see how risky the changes are.
We will simulate this situation in our example. In a new version of
the parser, the character classification functions in the code,
isWhiteSpace()
, isAlpha()
, etc., have been changed and
use the standard C classification functions, like isspace()
,
instead of strchr()
. The new version of the parser can be found
in the directory parser_v3
.
We will now compare it with the version in parser_v2
, but
neither run the tests nor even compile it. Instead we need the
following two pieces of information:
parser_v2
, as generated in the
previous section.parser
directory that you can use, parser.diff
.The diff file must be in the
“unified”
difference format. This is the standard output format of the
diff functionality of many version control systems, e.g. of
git diff
(see Chapter 36). Under
UNIX®-like systems, the patch file can also be generated by the
diff
utility. It would be invoked from the parser
directory in the following way:
There is also a Microsoft® Windows version of GNU diff in
the parser
directory, therefore the same command works in
a Windows command shell too.
Start the CoverageBrowser. Then load the instrumentation database
parser_v2/unittests.csmes
via the menu entry
"File->Open…", and the measurements file
parser_v2/unittests.csexe
with "File->Load
Execution Report…".
Now select the menu entry "Tools->Patch File Analysis…". When the "Patch File Analysis" dialog appears:
Then click "Open" to view the report in the browser.
The report consists of three tables that summarize the influence that the patch has on code coverage, and then an annotated version of the patch file.
The two tables in the section “Overview” contain statistics about the number and kind of the lines that were influenced by the patch.
The first table groups the patched lines in the code by the results of the tests that executed them. One can see here how much influence the patch has on tests that have passed (and now could fail) or failed (and could now succeed). There are also entries for manually checked tests and for those whose status is unknown. In our example, we did not register the test results and all our tests are counted as “Unknown”.
The second table shows the kind of changes to expect in the test coverage after the patch has been applied. It consists of three columns, containing the statistics about removed and inserted lines and their sum. From the first two columns one can see whether the test coverage for the patched code grows or falls. (In the parser example, it stays the same.) The last line in the table is also important: It shows the number of lines which Squish Coco could not classify as inserted or removed. Patch analysis is a heuristics, after all.
The section “List of tests influenced by the modifications” is a list with the names of the tests that executed the patched code, together with their result. It is helpful for a qualitative analysis of the patch. In our example, we can see that all tests execute code that is affected by the patch.
![]()
Figure 3.1: Coverage of the patched lines by the tests
More details can be found in the “Patch File” section of the report. It is an annotated version of the original patch file, with the old version of the text in red and the new version in green. Lines that did not change are shown in gray. The most important column is “Tests”, which shows for each code line the number of tests that executed it (if it is removed) or will probably execute it after the patch is applied. A tooltip shows the names of these tests.
Now suppose that a patch has already been checked in. It has changed the application’s behavior, and you want to write more test to make sure that all new code has been covered. In this case, a reverse patch analysis is helpful.
To simulate a reverse patch analysis with the parser example, switch
to the directory parser_v3
and compile the code there in the
same way as before parser_v2
. Then start the
CoverageBrowser and load this time the instrumentation database
parser_v3/unittests.csmes
and the measurements file
parser_v3/unittests.csexe
. This can also be done on the
command line, e.g. in the form
Generate the patch analysis report as before. Squish Coco recognizes
automatically that patch.diff
is a patch to the
current version of the program, and generates a corresponding report.
This patch report contains the same data as the report for the forward
patch, but the headers have changed to reflect the new interpretation.
CoverageBrowser can compute the probable location of an error in the source code by comparing the coverage data of all tests that were run on a program.
The algorithm expects that there at least one test had failed. In our example, the parser does not detect correctly that the exponent of a floating point number was invalid. It is possible to write the following:
But 1E+
is not a valid number because it has no digits after
the +
.
To find out which line in the source code causes this, we need a few
additional tests. The following tests are already part of the test
suite in the file unittests.cpp
:
Test input | Expected result | Execution status |
1.1 | 1.1 | Passed |
1.1e1 | 11 | Passed |
1.1e+1 | 11 | Passed |
1.1e-1 | 0.11 | Passed |
1.1e+ | error message | Failed |
With these tests already contained in the test suite, we can use CoverageBrowser to find a candidate for the line that caused the bug. There is a "Bug Location" window in the CoverageBrowser, but it is by default disabled. You can enable it by checking the field "Bug Location" in the menu "View->Bug Location".
After the window is enabled, select all the executions in the
"Executions" window and click the "Compute" button.
The window will then be filled with a list of possible bug locations,
with the most probable (according to Squish Coco) first. In our case there
is only one candidate: the line 263 of the file parser.cpp
.
This line is the location where the sign ’+
’ of the exponent is
parsed.
At this point a fix could be implemented, but since this is only a demonstration, we will not do it here.
The following example is more complex. We will take Qt’s TextEdit example and use it to illustrate how Squish Coco can be used at different stages of the development process. To cover the whole coding cycle, we will first show how an instrumented application is created, perform manual tests and analyze their results. Then we will create an instrumented unit test.
In a second step (see Chapter 4.2), we will cover the aspects which are more interesting to product managers: analyzing the impact of code changes (e.g. bug fixes)—in particular, tracking their test progress, externalizing testing, and collecting the code coverage analysis of a complete testing team.
The example can be found in a
directory called “textedit
”. Its location varies according
to the operating system. Under Microsoft® Windows, the directory is
stored directly in the installation directory. Under Linux™, it can be
found under /opt/SquishCoco/samples
, and under macOS it
is in /Applications/SquishCoco/samples
.
The example is write-protected, so you need to create a copy of the
directory textedit
to work with it.
In this section, we will work with the files in the directory
textedit/textedit_v1
.
We want to be able to build our application both normally and with generated test coverage instrumentation code, without having to change our source code. This can be achieved by making a small change to the application’s project (.pro) file. We can then use a command line option for qmake to generate an instrumented build instead of a normal one.
To make the project file suitable both for normal and for instrumented builds, we create a set of definitions that can be activated by a command line switch; in qmake’s terminology this is called a scope. The following listing (see Figure 4.1) shows a minimal scope for code instrumentation.
The following must be done:
PRECOMPILED_HEADER
variable to an empty value.QMAKE_LINK_OBJECT_MAX
in order to disable the usage of the
linker script. We set it here to 10000.CodeCoverage {
PRECOMPILED_HEADER=
QMAKE_LINK_OBJECT_MAX=10000
QMAKE_CC=cs$$QMAKE_CC
QMAKE_CXX=cs$$QMAKE_CXX
QMAKE_LINK=cs$$QMAKE_LINK
QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB
QMAKE_AR=cs$$QMAKE_AR
QMAKE_LIB=cs$$QMAKE_LIB
}
These modification are sufficient for most standard C and C++ applications. For Qt applications we must use additional settings in order to ensure that CoverageScanner does not instrument the source code that is generated by Qt’s tools (e.g., by uic, qrc and the moc).
To exclude qrc resource files from instrumentation, we must tell CoverageScanner not to instrument any file with a name that begins with qrc_. This can be done with the command line option --cs-exclude-file-abs-wildcard=*/qrc_*. Since we do not want to have to enter this option manually, we will put it in the .pro file. Similarly, to let CoverageScanner ignore the files generated by uic we can use the same command line option, only this time with a different file matching regular expression: --cs-exclude-file-abs-wildcard=*/ui_*.
Squish Coco also supports by default applications that are built with the Qt 4 toolkit.1 This ensures that CoverageScanner:
Q_OBJECT
and
Q_DECLARE_PLUGIN
macros, and
We can also exercise some control over the level of instrumentation and what information is reported. For example, we can switch on the counting of code executions with the --cs-count command line option, or we can enable full instrumentation at decision/condition level with the --cs-full-instrumentation option. With the --cs-output option we can specify the file the execution report is written to when the application terminates. (By default the output is written to the file ⟨appname⟩.csexe, where ⟨appname⟩ is the name of the program that has been executed.)
So, for a Qt 4-based application, the final Squish Coco scope in the application’s .pro file will look something like this:
CodeCoverage {
COVERAGE_OPTIONS = --cs-output=textedit.exe
COVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=*/qrc_*
QMAKE_CFLAGS += $$COVERAGE_OPTIONS
QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS
QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC
QMAKE_CXX=cs$$QMAKE_CXX
QMAKE_LINK=cs$$QMAKE_LINK
QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB
QMAKE_AR=cs$$QMAKE_AR
QMAKE_LIB=cs$$QMAKE_LIB
}
In this form, the scope has been added to the project file
textedit_v1/textedit_v1.pro
. When we now run qmake without
options, a Makefile for a normal build is generated; but we
can also build an instrumented version of the program in the following
way:
In either case we still end up with a textedit.exe
executable. But with an instrumented build we will also get an
additional file, textedit.exe.csmes
. It contains the
instrumentation database.
For our very first exercise we will simply execute TextEdit and
then quit the application straight away. This will generate a file
called textedit.exe.csexe
. The file is in a binary format,
so it is not human readable. It is an instrumentation database that
contains a snapshot of the most recent execution that we have just
done.
To see the results we must run the CoverageBrowser tool and load the
textedit.exe.csmes
instrumentation database. This can be
done with the menu entry "File->Open…". After the file
has been opened, no coverage information is available because no
execution reports have been imported. The instrumented code lines are
shown grayed out and no coverage statistics has been computed
(see Figure 4.3).
![]()
Figure 4.3: CoverageBrowser after loading the TextEdit’s instrumentation database
In order to see an execution report, click "File->Load
Execution Report…", which invokes the import dialog. You need to
enter at least the file name (including the full path) of the
textedit.exe.csexe
file into the field "File Name".
You also need to give the test a name. Enter in the field
"Name" e.g. the text “Start and Quit”. Switch the
"Delete after loading" option on because the report is no
longer needed after our import. It is also helpful to set the option
"When file becomes modified" to “Open this dialog”, because
then the file import dialog is automatically opened after each run.
The new .csexe file that it created in the test can then be
added to the database.
After the import has finished, the code coverage information is visible:
CoverageBrowser shows reveals, for example, that the
TextEdit::fileSave()
function is not executed. We will now
validate this function interactively, guided by the code coverage
analysis.
In the source window, all unexecuted source code lines are shown on a red background (see Listing 4.4).
bool TextEdit::fileSave()
{
if (fileName.isEmpty())
{
QMessageBox::warning(this, tr("No file name specified"),
tr("Save first your document using 'Save As...' from the menu"),
QMessageBox::Ok);
return false;
}
QTextDocumentWriter writer(fileName);
bool success = writer.write(textEdit->document());
if (success)
textEdit->document()->setModified(false);
return success;
}
Figure 4.4: CoverageBrowser source view of the function TextEdit::fileSave()
To test this function, we must perform the following steps:
After these steps have been done and the coverage report imported,
CoverageBrowser shows that the “return false;
” line
just after the call to QMessageBox::warning()
has been
executed, as indicated by the green background). However, the line
“if (fileName.isEmpty())
” is partially executed: This is
shown by an orange background (see Listing 4.5).
bool TextEdit::fileSave()
{
if (fileName.isEmpty())
{
QMessageBox::warning(this, tr("No file name specified"),
tr("Save first your document using 'Save As...' from the menu"),
QMessageBox::Ok);
return false;
}
QTextDocumentWriter writer(fileName);
bool success = writer.write(textEdit->document());
if (success)
textEdit->document()->setModified(false);
return success;
}
Figure 4.5: CoverageBrowser source view after clicking TextEdit’s ‘Save’ button.
The explanation window (see Listing 4.6) tells us that the value of
the expression “fileName.isEmpty()
” was true during one
execution but was never false. It is therefore considered as only
partially executed. In order to fully test this expression, we must
click on the ‘Save As…’ button, then choose a file name,
and finally click on the ‘Save’ button.
After rerunning the application and doing a “Save as”, the new
execution report has only one source code line that is partially
untested (see Listing 4.7). In this case, CoverageBrowser shows
that the Boolean variable success
was never false, which means
that saving the document has never failed.
We could force a write failure, and this would ensure that we had 100% code coverage for this function. But we will use a different test strategy to get complete code coverage: We will use a unit test and import the execution result into the TextEdit instrumentation database.
bool TextEdit::fileSave()
{
if (fileName.isEmpty())
{
QMessageBox::warning(this,tr("No file name specified"),
tr("Save first your document using 'Save As...' from the menu"),
QMessageBox::Ok );
return false;
}
QTextDocumentWriter writer(fileName);
bool success = writer.write(textEdit->document());
if (success)
textEdit->document()->setModified(false);
return success;
}
Figure 4.7: CoverageBrowser source view after clicking TextEdit’s ‘Save As…’ button and then the ‘Save’ button.
The unit test infrastructure can be found in the directory
textedit_v1_tests/
. It contains just one test, which sets an
illegal filename and then tries to execute TextEdit’s
fileSave()
function. To do this, we use the QTestLib unit
test library that is part of Qt. The test is contained in the file
textedit_v1_tests/tst_textedit.cpp
(see Listing 4.8).
#include "tst_textedit.h"
void TestTextEdit::tst_saveFile() {
TextEdit textEdit;
textEdit.fileName="/";
QVERIFY( !textEdit.fileSave() );
}
QTEST_MAIN(TestTextEdit);
Figure 4.8: Unit test for the TextEdit application, tst_textedit.cpp
To import the instrumentation result of this test into TextEdit’s instrumentation database, the following infrastructure is necessary:
The unit test will recompile along with
textedit_v1/textedit.cpp
. To make its results importable
into the TextEdit instrumentation database, it is necessary that
both executables, TextEdit and the unit test, are instrumented in
exactly the same way. Since we use only the default
instrumentation, this requirement is already fulfilled.
Unfortunately, this is not everything we need to do. Squish Coco’s default behavior is to instrument only header and source files in the directory it is invoked, but here we need to instrument the TextEdit application’s sources in addition to the unit test. Therefore we must use another command line option to specify an additional path for files to instrument: --cs-include-path.
As before, we do not want to have to remember these command line
arguments every time, so we set them in the qmake project file,
textedit_v1_tests.pro
(see Figure 4.9). With these
lines in the unit test project file, the qmake-generated
Makefile will create the tst_textedit.exe
executable which, when run, produces the execution report
tst_textedit.exe.csexe
. We then can use the
CoverageBrowser to import this report into the file
tst_textedit.exe.csmes
.
HEADERS = ../textedit_v1/textedit.h tst_textedit.h
SOURCES = ../textedit_v1/textedit.cpp tst_textedit.cpp
CodeCoverage {
COVERAGE_OPTIONS = --cs-output=tst_textedit.exe
COVERAGE_OPTIONS += --cs-include-path=../textedit_v1
COVERAGE_OPTIONS += --cs-exclude-file-abs-wildcard=*/qrc_*
QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS
QMAKE_CCFLAGS += $$COVERAGE_OPTIONS
QMAKE_LFLAGS += $$COVERAGE_OPTIONS
QMAKE_CC=cs$$QMAKE_CC
QMAKE_LINK=cs$$QMAKE_LINK
QMAKE_CXX=cs$$QMAKE_CXX
}
We can also execute the unit test automatically and import the
execution report with a post-build rule. Squish Coco provides an
extra command line tool, cmcsexeimport (see Chapter 24),
which imports an execution report into an instrumentation database.
The post-build rule first deletes any previous execution report, then
executes the test itself, and finally imports the results into the
application’s execution database, tst_textedit.exe.csexe
(see Listing 4.10).
CodeCoverage {
win32: MAINDIR=$$replace(PWD,"/","\\")
!win32:MAINDIR=$$PWD
unix {
QMAKE_POST_LINK = rm $$MAINDIR/tst_textedit_v1.exe.csexe ;
QMAKE_POST_LINK += $$MAINDIR/tst_textedit_v1.exe ;
QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR/tst_textedit_v1.exe.csmes \
-e $$MAINDIR/tst_textedit_v1.exe.csexe -t UnitTest
}
win32 {
QMAKE_POST_LINK = del /F $$MAINDIR\\tst_textedit_v1.csexe &
QMAKE_POST_LINK += $$MAINDIR\\tst_textedit_v1.exe &
QMAKE_POST_LINK += cmcsexeimport -m $$MAINDIR\\tst_textedit_v1.exe.csmes \
-e $$MAINDIR\\tst_textedit_v1.csexe -t UnitTest
}
}
Figure 4.10: Post-build rules for the import of the execution report into the unit test’s instrumentation database
By default, the coverage data is imported without any information about the executed tests. Instead, an execution report called ‘UnitTest’ is created, which does not describe which test was executed or whether its execution was successful. To provide the missing information, we must use the CoverageScanner API and generate an execution report for each test that is executed. An example that shows how this is done is available in the chapter 38.2. In the example, the API is used in the following way:
HEADERS += testcoverageobject.h
SOURCES += testcoverageobject.cpp
Figure 4.11: Including the CoverageScanner listener in textedit_v1_tests.pro
#include "testcoverageobject.h"
#include "../textedit_v1/textedit.h"
#include <QtTest/QtTest>
class TestTextEdit : public TestCoverageObject
{
Q_OBJECT
private slots:
void tst_saveFile();
};
Figure 4.12: The TextEdit unit test header file, tst_textedit.h
The source code of the file testcoverageobject.cpp
(see Listing 4.13) is simple to understand. The file adds a single
cleanup()
function to QTestLib, which is executed after each
unit test item. The code between
#ifdef __COVERAGESCANNER__
and #endif
is only
compiled when CoverageScanner is invoked2.
This extra code creates a test name by combining the test class’s
object name and the test function’s name. In the the example, an
execution item called unittest/TestTextEdit/tst_saveFile
is
generated. The slash is used to support the organization of tests in a
tree view. The current test status (“PASSED” or “FAILED”) is also
recorded and added to the execution report by the
__coveragescanner_save()
function.
...
void TestCoverageObject::cleanup()
{
cleanupTest();
#ifdef __COVERAGESCANNER__
QString test_name="unittest/";
test_name += metaObject()->className();
test_name += "/";
test_name += QTest::currentTestFunction();
__coveragescanner_testname(test_name.toLatin1());
if (QTest::currentTestFailed())
__coveragescanner_teststate("FAILED");
else
__coveragescanner_teststate("PASSED") ;
__coveragescanner_save();
#endif
}
Figure 4.13: An extract from the TestCoverageObject’s source code
At this stage we could start CoverageBrowser, load the TextEdit instrumentation database, and import the unit test’s instrumentation database by clicking "File->Import Unit Tests…". An even more convenient alternative is to use the cmmerge tool to automate this step. The cmmerge program is designed to import one instrumentation database’s execution report into another instrumentation database. This means that we can extend our post-build rules to use the cmmerge program to import the coverage information automatically from the unit test into the TextEdit program’s instrumentation database (see Listing 4.14).
CodeCoverage {
# Merge coverage database into TextEdit database
unix {
QMAKE_POST_LINK += ;
QMAKE_POST_LINK += cmmerge -o $$MAINDIR/../textedit_v1/textedit.tmp \
-i $$MAINDIR/../textedit_v1/textedit.exe.csmes \
$$MAINDIR/./tst_textedit_v1.exe.csmes &&
QMAKE_POST_LINK += rm $$MAINDIR/../textedit_v1/textedit.exe.csmes &&
QMAKE_POST_LINK += mv $$MAINDIR/../textedit_v1/textedit.tmp \
$$MAINDIR/../textedit_v1/textedit.exe.csmes
}
win32 {
QMAKE_POST_LINK += &
QMAKE_POST_LINK += echo Merging unit test result into the main application &
QMAKE_POST_LINK += cmmerge -o $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \
-i $$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes \
$$MAINDIR\\tst_textedit_v1.exe.csmes &
QMAKE_POST_LINK += COPY /Y $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes \
$$MAINDIR\\..\\textedit_v1\\textedit.exe.csmes &
QMAKE_POST_LINK += DEL /F $$MAINDIR\\..\\textedit_v1\\textedit_unit.exe.csmes
}
}
Figure 4.14: Post build rules: merging instrumentation results into the TextEdit instrumentation database
With all these changes to the .pro file in place, we can once
again build and run the unit test. Now the CoverageBrowser shows the
fileSave()
function to be 100% covered, with the execution list
containing our three original manual tests and the single unit test
(see Figure 4.15).
![]()
Figure 4.15: The execution list after all the tests have been executed
The most common ways in which code coverage is used are for developers to use it to find untested code, and for managers to use it to produce test status reports (e.g., as diagrams).
In addition to fully supporting the common use cases, Squish Coco also provides additional features which make it possible to go beyond these fundamentals and extend what can be achieved with code coverage. This will be discussed in the current subsection.
Recording each test’s coverage data makes it possible to compare their data to answer the question, “What does this test cover that the others do not?” This is particularly useful if just one test fails, since it can help us to identify which part of the code is involved.
To see how this works in practice, let us return to the TextEdit example. If we click ‘Save’, we will get an error message that no file name is defined. This is not very convenient for users—we should have designed TextEdit to handle this particular case by opening the ‘Save As…’ dialog rather than by producing an error.
To identify where in the code this problem arises, we simply compare the ‘Save Clicked’ execution with all other executions that involve the ‘Save’ button. To do this we must first switch to the “Execution Comparison Analysis” mode (Menu: "Tools->Execution Comparison Analysis"). Select the checkboxes in the “Reference” column for the “tst_saveFile” and “SaveAs clicked before Save clicked” tests. This will make the execution comparison symbol appears in front of the affected names (see Figure 4.16). In the “Executions” column, click on “Save clicked” checkbox.
![]()
Figure 4.16: The execution list being used to compare different executions
In this mode, the coverage analysis is based only on source code lines which are not executed by “tst_saveFile” and “SaveAs clicked before Save clicked”. This is why the overall coverage decreases to 1.29%: It means that “Save clicked” executes 1.29% more code than the selected tests.
Using cmreport it is possible to generate a HTML report which displays the same information:
If we now look at the source code itself, we will see that only two lines
of the TextEdit::fileSave()
function (see Listing 4.17) are
not grayed: the lines which pop up the error message. These are the
lines that must be modified to change the “Save” button’s
behavior.
bool TextEdit::fileSave()
{
if (fileName.isEmpty())
{
QMessageBox::warning(this,tr("No file name specified"),
tr("Save first your document using 'Save As...' from the menu"),
QMessageBox::Ok );
return false;
}
QTextDocumentWriter writer(fileName);
bool success = writer.write(textEdit->document());
if (success)
textEdit->document()->setModified(false);
return success;
}
Figure 4.17: CoverageBrowser source view of the comparison of the “Save clicked” execution.
In this case, changing
the fileSave()
function is easy—we simply replace the
QMessageBox::warning()
call with a call to the
fileSaveAs()
method.
(see Listing 4.18)
bool TextEdit::fileSave()
{
if (fileName.isEmpty())
return fileSaveAs();
QTextDocumentWriter writer(fileName);
bool success = writer.write(textEdit->document());
if (success)
textEdit->document()->setModified(false);
return success;
}
Figure 4.18: The TextEdit’s improved fileSave() function.
Before committing a change or starting to test a hot fix, it is possible to estimate the impact of the code modification. CoverageBrowser is able to perform an analysis on the difference between two source sets and can list the tests that will be affected (and those which will not).
Start CoverageBrowser and load the modified TextEdit example’s instrumentation database. Now click on "Tools->Compare with…" and select the original version of the TextEdit instrumentation database. CoverageBrowser will now display the source code like a text comparison application does (e.g., diff).
Click on "Tools->Analysis of Modified Methods" to exclude all
unmodified functions from the coverage analysis. In the TextEdit case,
doing this will mean that only one function,
TextEdit::fileSave()
, will be treated as being instrumented since
that is the only method we have changed
(see Figure 4.19). This also affects the statistic calculations
since execution coverage statistics will now be limited to just this
function. The test executions whose coverage statistic is not
zero are the ones that are affected by the code modifications we have made.
In our case we have:
The “Start and Exit” case has a coverage of 0% and so does not execute our modified code. It is for this reason no longer visible in the execution list. All entries in the ‘Execution’ column are struck through to inform us that these tests are not executed in the newest version and only present in the reference database.
In other words, only the two manual tests and the unit test listed above must be re-executed to ensure that no regressions have been introduced by our code changes.
![]()
Figure 4.19: List of tests affected by a code modification
Up to now we have done white-box testing, i.e. testing where we have access to the source code and which makes use of our knowledge of the code. It is also possible to use Squish Coco for black-box testing. In other words, we can still do code coverage analysis without having access to or even knowledge of the source code. If we use this approach, the generated instrumentation database will, of course, contain no source code.
To use black-box testing we must create a suitable instrumentation database by clicking "File->Generate Black-Box Configuration…". This database, along with the TextEdit executable, can be given to the test team which can then use them with a simplified version of CoverageBrowser (see Figure 4.20). This version of CoverageBrowser only supports the importing and managing of execution reports since it does not have access to the application’s source code.
![]()
Figure 4.20: Black-box testing results as shown by CoverageBrowser
Once all the tests are finished, the black-box database can be merged into the original TextEdit instrumentation database using CoverageBrowser’s merge facilities (Menu: "File->Merge with…").
Often, when a small bug fix is made, the effects are very localized and leave most of the source code unchanged. In view of this, it is often unnecessary to retest the entire application with the whole test suite.
Squish Coco makes it possible to avoid unnecessary testing. We can tell it to restrict itself to the source code that has changed between the original and fixed version of the application (see Chapter 4.2.2). This allows us to focus purely on the analysis of the fix. To achieve this, simply load the fixed application’s freshly generated instrumentation database (e.g., for the modified TextEdit application), and compare it with the earlier database for the unfixed version, using Squish Coco’s facility for analyzing modified functions.
![]()
Figure 4.21: Coverage of the patched function
We have done just such a comparison and the results are shown in in
4.21: The two tests, “Save clicked” and “Start
and Exit”, cover 85% of the TextEdit::fileSave()
function, the
only method that was modified for our fix. From this we know exactly
what additional testing is necessary to achieve 100% code coverage for
our tests for the fixed version of the application. CoverageBrowser continues to display
the list of missing tests (which are only executed using the first
version of TextEdit) in strikeout style.
This gives a hint of what remains for testing effort.
Using cmreport it is possible to generate a HTML report which displays the same information:
Squish Coco provides code coverage analysis which can be applied to all the usual testing techniques: unit, manual, and black-box testing. Squish Coco can easily be told to ignore the generated code produced by the Qt library’s tools (moc, qrc, and uic), so that only the code written by developers is instrumented. Test results can be collected into a database and can be used to evaluate how much code coverage our tests achieve and to show which statements are not currently tested. With this information we can target our testing efforts towards 100% test code coverage. In addition, Squish Coco makes it possible to see what effect a code modification would have in terms of test code coverage without having to test the entire application.
Overall, Squish Coco can help us target our tests to ensure that our applications have as much test coverage as possible, while avoiding or minimizing test coverage duplication. Furthermore, Squish Coco can help us see what effects changes to our code have on test coverage, so that we can adapt our test suites accordingly.