Squish Coco
Code Coverage Measurement for C/C++
Part II
|
| Linux™/Apple® Mac OS X | Microsoft® Windows |
qmake CONFIG+=CodeCoverage make | qmake CONFIG+=CodeCoverage nmake |
Before modifying the project file, we must ensure that precompiled
headers are disabled. This is a compiler-specific feature so how to
switch them off depends on the compiler being used. qmake permits to deactivate
the usage of it by simply emptying the PRECOMPILED_HEADER variable.
Also it is necessary to increase the value of QMAKE_LINK_OBJECT_MAX to
disable the usage of linker script.
To make the .pro file suitable for doing both normal and instrumented builds, we must create a CodeCoverage variable that contains the settings we want to use. (This is the variable that we used on the command line shown above.) In particular, we must change the compiler and linker to use CoverageScanner’s wrappers rather than to use the native tools directly. This is easily done by prefixing the tools’ names with cs. Here’s an example of the modification needed for the .pro file:
This should be sufficient for most standard C and C++ applications, but for Qt applications we must use some additional settings in order to ensure that CoverageScanner doesn’t instrument the source code that is machine-generated by Qt’s tools (e.g., by uic, qrc and the moc).
To exclude instrumenting qrc resource files, we must tell CoverageScanner not to instrument any file whose name begins with qrc_. This can be done by using the command line option --cs-exclude-file-regex=qrc_.*. Of course, we don’t want to have to enter this option manually, so we will put it in the .pro file. Similarly, to make CoverageScanner ignore the files generated by uic we can use the same command line option, only this time with a different file matching regular expression: --cs-exclude-file-regex=ui_.*.
Squish Coco also provides a command line option that is specific to
applications built using the Qt 4 toolkit:
(--cs-qt4). This option ensures that CoverageScanner:
-
does not instrument the
Q_OBJECTandQ_DECLARE_PLUGINmacros. - does not instrument code generated by the moc, except that signal emissions and slot receives are instrumented, since these are usually vital to a Qt program’s logic.
We can also exercise some control over the level of instrumentation and
what information is reported. For example, we can switch on counting of
code executions using the --cs-count command line option, or we
can switch on a full instrumentation at decision/condition level using
the --cs-full-instrumentation command line option. We can also
specify which file the execution report should be written to when the
application terminates using the --cs-output command line option
(by default the output is written to the file coverage.csexe).
So, for a Qt 4-based application, the final Squish Coco configuration in the application’s .pro file will look something like this:
With this in place we can use qmake to create a Makefile for a normal build or for an instrumented build. In either case we still end up with a textedit.exe executable, but when we do an instrumented build we will also get an additional file, textedit.exe.csmes, which contains the instrumentation database.
The First Code Coverage Results
For our very first exercise we will simply execute TextEdit and then quit the application straight away. This will cause a file called textedit.exe.csexe to be generated. This file is in a binary format, so it isn’t human readable—it is an instrumentation database that contains a snapshot of the most recent execution (i.e., the one we have just done).
To be able to see the results we must run the CoverageBrowser tool and load the textedit.exe.csmes instrumentation database (Menu: "File->Open…"). When the file is been opened, no coverage information is available because no execution reports have been imported. The instrumented code lines are shown grayed-out and no coverage statistics have been computed. (see figure fig:startup)
Figure 4.1: CoverageBrowser after loading the TextEdit’s instrumentation database
To get an execution report click "File->Load Execution Report…" to invoke the import dialog. As a minimum you should at least enter the filename (including the full path) of the textedit.exe.csexe file (field: ‘File Name’), and give the test a name (Field: ‘Name’)—for example, “Start and Quit”. It is best to switch on the ‘Delete after loading’ option since the report is no longer needed after our import. It is also helpful to set the ‘When file becomes modified’ option to “Open this dialog” so that this dialog is opened automatically after each test execution.
Once the import has finished, the coverage information is now visible:
- The coverage statistics for each source file’s functions and methods for the whole application are shown.
- The source window is now colored and shows executed code on a green background and unexecuted code on a red background.
- The execution list now contains one selected item called “Start and Quit” which is the only test execution report we have created so far.
Interactive testing
CoverageBrowser correctly reveals, for example, that the
TextEdit::fileSave() function is not executed. We will try to
validate this function interactively, guided by the code coverage
analysis.
In the source window, all unexecuted source code lines are shown with a red background. (see listing 4.3.2)
To test this function we must perform the following steps:
- Start the TextEdit application;
- click on the ‘Save’ button: TextEdit should display the error message "Save first your document using ‘Save As…’ from the menu";
- quit the application.
After these steps have been done and the coverage report imported,
CoverageBrowser now shows that the return false; line just
after the call to QMessageBox::warning() has been executed (as
indicated by the green background). However, the line
if (fileName.isEmpty()) is shown as partially executed (indicated
with an orange background); (see listing 4.3.2).
The explanation window (see listing 4.3.2) tells us that
the evaluation of the expression fileName.isEmpty() was true
during one execution but was never false (hence, it is considered only
partially executed). In order to fully test this
expression we must click on the ‘Save As…’
button, then choose a filename, and finally, click on the ‘Save’
button.
After rerunning the application and doing a “save as”, the new
execution report now has only one source code line that is partially
untested (see listing 4.3.2). In this case, CoverageBrowser reveals
that the Boolean variable success was never false, that is,
saving the document has never failed.
Of course we could force a write failure, and this would ensure that we have 100% code coverage of this function. But we will use a different test strategy to get complete code coverage: we will implement a unit test and import the execution result into the TextEdit instrumentation database.
Writing unit tests
We will write a unit test which sets an illegal filename, and then tries
to execute the TextEdit’s fileSave() function. To do this we
will use the QTestLib unit testing library that is supplied with Qt as
standard (see listing 4.3.2).
To get this test’s instrumentation result into the TextEdit’s own instrumentation database, we must perform the following steps:
- Create a qmake project file with code coverage configured identically as for the TextEdit project.
- Add a post build rule which automatically executes the test and collects the coverage information.
- Add a unit test listener which saves the code coverage data (and the test status—passed or failed) for each unit test that’s executed into the unit test’s own instrumentation database.
- Import the code coverage report into the TextEdit’s instrumentation database.
The unit test will recompile itself, along with textedit.cpp.
To be importable into the TextEdit instrumentation database it is
necessary that both executables (TextEdit and the unit test) are
instrumented in exactly the same way. So, for this example, we
must first use the same instrumentation options: --cs-count,
--cs-full-instrumentation and --cs-qt4.
Unfortunately, these command line options alone are insufficient, because
Squish Coco’s default behavior is to only instrument header and source
files in the current directory. But here we need to instrument the
TextEdit application’s sources in addition to the unit test, so we
must use another command line option to specify an additional path for
files to instrument: --cs-include-path.
As before, we don’t want to have to remember these command line
arguments every time, so instead we set them in the qmake project
file as shown in Figure 4.3.2. With these lines added to the
unit test’s .pro file, the result will be that the
qmake-generated Makefile will create the
tst_textedit.exe executable which when run produces the
tst_textedit.exe.csexe execution report. We can then use the
CoverageBrowser to import this report to produce the
tst_textedit.exe.csmes file.
We can automatically execute the unit test and import the execution
report using a post build rule. Squish Coco provides an extra command
line tool called cmcsexeimport which imports an execution report into
an instrumentation database. The post build rule must begin by deleting
any previous execution report, then it must execute the test itself, and
finally it must import the results into the application’s execution
database (in this case the tst_textedit.exe’s execution database).
(see listing 4.3.2).
By default the coverage data is imported without any information about the executed tests: instead, an execution called ‘UnitTest’ is created which does not describe which test was executed or even if its execution was successful. To provide the missing information we must use the CoverageScanner API to generate an execution report for each test that is executed. An example that shows how this is done is available in the chapter 26.2.4. Using the API is really easy:
- Add two Squish Coco source files to the qmake project file (see listing 4.3.2).
- Change the unit test to inherit from TestCoverageObject instead of from QObject (see listing 4.3.2).
The testcoverageobject.cpp file’s source code
(see listing 4.3.2) is simple to understand. The file adds a single
cleanup() function to QTestLib, which is executed after each
unit test item. The code between #ifdef __COVERAGESCANNER__ and
#endif is only compiled when CoverageScanner is
invoked1. This extra code creates a test name by combining the test
class’s object name and the test function’s name. In this case, an
execution item called unittest/TestTextEdit/tst_saveFile is
generated. The slash is used to support the orgainization of tests in a
tree view. The current test status (“PASSED” or “FAILED”) is also
recorded, and added to the execution report by calling the
__coveragescanner_save() function.
At this stage we could start CoverageBrowser, load the TextEdit instrumentation database, and import the unit test’s instrumentation database by clicking "File->Import Unit Tests…". An even more convenient alternative is to use the cmmerge tool to automate this step. The cmmerge program is designed to import one instrumentation database’s execution report into another instrumentation database. This means that we can extend our post build rules to use the cmmerge program to automatically import the coverage information from the unit test into the TextEdit program’s instrumentation database (see listing 4.3.2).
With all these changes to the .pro file in place, we can once
again build and run the unit test. Now the CoverageBrowser shows the
fileSave() function to be 100% covered, with the execution list
including our three original manual tests and the one unit test
(see figure fig:execlist1).
Figure 4.2: The execution list after all the tests have been executed
4.3.3 Working with code coverage data
The commonest ways that code coverage is used are for developers to use it to find untested code, and for managers to use it to produce test status reports (e.g., as diagrams).
In addition to fully supporting the common use cases, Squish Coco also
provides additional features which make it possible to go beyond these
fundamentals to extend what can be achieved with code coverage, as we
will discuss in this subsection.
Post mortem analysis
Recording each test’s coverage data makes it possible to compare their data to answer the question: “what does this test cover that the others don’t?”. This is particularly useful if just one test fails, since it can help us to identify which part of the code is involved.
To see how this works in practice, let’s return to the TextEdit example. If we click ‘Save’ we will get an error message saying that no filename is defined. This is not very convenient for users—really we should have designed TextEdit to handle this particular case by opening the ‘Save As…’ dialog rather than by producing an error.
To identify where in the code this problem arises, we simply compare the ‘Save Clicked’ execution with all other executions that involve the ‘Save’ button. To do this we must begin by changing to the “Execution Comparison Analysis” mode (Menu: "Tools->Execution Comparison Analysis"). Select in the “Reference” column for the “tst_saveFile” and “SaveAs clicked before Save clicked” tests. This will make the execution comparison symbol appears in front of the affected names (see figure fig:execlist2). In the “Executions” column, click on “Save clicked” execution.
Figure 4.3: The execution list being used to compare different executions
In this mode the coverage analysis is made only on source code lines which are not executed by “tst_saveFile” and “SaveAs clicked before Save clicked”. This is why the overall coverage decreases to 1.29% which means in this case that “Save clicked” executes 1.29% more code than the selected tests.
Using cmreport it is possible to generate a HTML report which displays the same information:
cmreport --csmes=textedit_v1/textedit.exe.csmes \
--html=textedit.html \
--global=all --execution=all --source=all --executions=all \
--toc --method=all --select=".*" \
--select-reference=".*tst_saveFile" \
--select-reference="SaveAs clicked before Save clicked"
If we now look at the source code itself we will see that only two lines
of the TextEdit::fileSave() function (see listing 4.3.3) are
not grayed: the lines which pop-up the error message. These are the
lines that must be modified to change the “Save” button’s
behaviour.
In this case, changing
the fileSave() function is easy—we simply replace the
QMessageBox::warning() call with a call to the
fileSaveAs() method.
(see listing 4.3.3)
Evaluating the impact of a hot fix
Before committing a change or starting to test a hot fix, it is possible to estimate the impact of the code modification. CoverageBrowser is able to perform an analysis on the difference between two source sets and can list those tests which will be affected (and those which won’t).
Start CoverageBrowser and load the modified TextEdit example’s instrumentation database. Now click on "Tools->Compare with…" and select the original version of the TextEdit instrumentation database. CoverageBrowser will now display the source code like a text comparison application does (e.g., diff).
Click on "Tools->Analysis of Modified Methods" to exclude all
unmodified functions from the coverage analysis. In the TextEdit case,
doing this will mean that only one function,
TextEdit::fileSave(), will be treated as being instrumented since
that is the only method we have changed
(see figure fig:patchimpact). This also affects the statistic calculations
since execution coverage statistics will now be limited to just this
function. Those test executions whose coverage statistic is not
zero are the ones affected by the code modifications we have made.
In our case we have:
- “Save clicked”,
- “SaveAs clicked before Save clicked” and
- “tst_saveFile” (our unit test).
The “Start and Exit” case has a coverage of 0% and so does not execute our modified code, and is for this reason no more visible in the execution list. All this execution are strikeout to inform that this tests are not executed in the newest version and only present in the reference database.
In other words, only the two manual tests and the unit test listed above must be re-executed to ensure that no regressions have been introduced by our code changes.
Figure 4.4: List of tests affected by a code modification
Black-box testing/distributed testing
Up to now we have done white-box testing, that is, testing where we have access to the source code and which makes use of our knowledge of the code. It is also possible to use Squish Coco for black-box testing, in other words, we can still do code coverage analysis without having access to or even knowledge of the source code. If we use this approach the generated instrumentation database will, of course, contain no source code.
To use black-box testing we must create a suitable instrumentation database by clicking "File->Generate Black-Box Configuration…". This database, along with the TextEdit executable, can be given to the test team which can then use them with a simplified version of CoverageBrowser (see figure fig:blackbox). This version of CoverageBrowser only supports the importing and managing of execution reports since it does not have access to the application’s source code.
Figure 4.5: Black-box testing results as shown by CoverageBrowser
Once all the tests are finished, the black-box database can be merged into the original TextEdit instrumentation database using CoverageBrowser’s merge facilities (Menu: "File->Merge with…").
Verifying if a bug fix is correctly tested
Often, when a small bug fix is made, the effects are very localized and leave most of the source code unchanged. In view of this, it is often unnecessary to retest the entire application with the whole test suite.
Squish Coco makes it possible to avoid unnecessary testing since we can tell it to restrict itself to the source code that has changed between the original and fixed version of the application2. This allows us to focus purely on the analysis of the fix. To achieve this, simply load the fixed application’s freshly generated instrumentation database (e.g., for the modified TextEdit application), and compare it with the earlier database for the unfixed version, using Squish Coco’s facility for analysing modified functions.
Figure 4.6: Coverage of the patched function
We have done just such a comparison and the results are shown in in
4.6: the two tests, “Save clicked” and “Start
and Exit” cover 85% of the TextEdit::fileSave() function, the
only method that was modified for our fix. From this we now know exactly
what additional testing is necessary to achieve 100% code coverage for
our tests for the fixed version of the application. CoverageBrowser continue to display
the list of missing tests (which are only executed using the first version of TextEdit) in strikeout style,
which gives a hint of what remains for testing effort.
Using cmreport it is possible to generate a HTML report which displays the same information:
cmreport --csmes=textedit_v2/textedit.exe.csmes \
--csmes-reference=textedit_v1/textedit.exe.csmes \
--html=textedit.html \
--global=all --execution=all --source=all --executions=all \
--toc --method=all --select=".*"
4.4 Conclusion
Squish Coco provides code coverage analysis which can be applied to all the usual testing techniques: unit, manual, and black-box testing. Squish Coco can easily be told to ignore the generated code produced by the Qt library’s tools (moc, qrc, and uic), so that only the code that is written by developers is instrumented. Test results can be collected into a database and can be used to evaluate how much code coverage our tests achieve, and to show which statements are not currently tested so that we can target our testing efforts to move towards 100% test code coverage. In addition, Squish Coco makes it possible to see what effect a code modification would have in terms of test code coverage without us having to test the entire application.
Overall, Squish Coco can help us target our tests to ensure that our applications have as much test coverage as possible, while avoiding or minimizing test coverage duplication. Furthermore, Squish Coco can help us see what effects changes to our code have on test coverage so that we can adapt our test suites accordingly.
| ©2011 froglogic GmbH |  squish@froglogic.com |