Menu

Squish Coco

Code Coverage Measurement for Tcl, QML, C# and C/C++

Part VI
Tcl Coverage

Chapter 14  Command Line Arguments

Syntax:

coveragescannertcl [⟨cs-options⟩] ⟨application arguments

Where:

application arguments
Tcl-based application including optional arguments
cs-options
Any of the following options:
--cs-output=STRING
Base name of the generated .csmes and .csexe files
--cs-exclude-path=STRING
Exclude files located in specified directory
--cs-include-path=STRING
Override exclusion of files based on specified directory
--cs-exclude-file-abs-regexp=STRING
Exclude file if absolute file name matches regular expression
--cs-exclude-file-abs-regexp=STRING
Override exclusion of file if absolute name matches regular expression
--cs-exclude-file-abs-wildcard=STRING
Exclude file if absolute name matches wildcard
--cs-exclude-file-abs-wildcard=STRING
Override exclusion of file if absolute name matches wildcard
--cs-include-code-regexp=STRING
Regular expression of code that will be instrumented even if it is excluded
--cs-exclude-code-regexp=STRING
Regular expression of code that will not be instrumented
--cs-include-code-wildcard=STRING
Wildcard of code that will be instrumented even if it is excluded
--cs-exclude-code-wildcard=STRING
Wildcard of code that will be instrumented even if it is excluded
--cs-include-init-scripts
Enables code instrumentation for standard Tcl initialization scripts
--cs-include-unnamed
Instrument Tcl code even if it has no or just an ambiguous file name set. By default such code is ignored as it is typically just temporary code generated for e.g. an Tk event handler or results from nested eval calls.
@path
Read command line options from the file at ⟨path⟩ and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.

CoverageScannerTcl is a utility which permits to collect code coverage information of a Tcl-based execution. It will collect all scripts parsed by the embedded Tcl interpreter and determine which parts were actually executed.

Once the application exits a pair of .csmes and .csexe files will be written out that contain the code instrumentation information and execution data, respectively.

If the --cs-output switch is omitted the base name of the output files will the same as the ⟨application⟩ name.

See (see Chapter 15) for a recommended set of --cs-exclude-code-regexp switches for typical Tcl/Tk applications.

Chapter 15  Mixed Tcl/C/C++ Code Coverage Analysis

15.1  Analysis of Tcl script coverage

15.1.1  Demo application build

Copy the sample to your working diretcory

cp -r /opt/SquishCoco/doc/demoapp_tcl ./
cd demoapp_tcl

Edit Makefile and adjust the following variables

TCLLIBS=-ltk8.5 -litcl3.4 -ltcl8.5
TCLINCLUDES=-I/usr/include/tcl8.5

to match your system’s library names and include paths.
Build the C++ executable with:

$ make

Test the simple Tk GUI with an invocation of the launcher script:

$ ./proto

15.1.2  Ad-hoc Tcl coverage run

Launch the application under test via the Tcl coverage scanning tool :

coveragescannertcl --cs-output=tcl ./proto

The GUI will show up as normal while code coverage is monitored. At this point the application might already be shut down via the "File->Exit" menu entry or the [X] button in the window’s title bar. Observe the two files tcl.csmes and tcl.csexe being generated. The latter contains data about the code covered in the last execution. The former is a measurement database containing information about the instrumented code and possibly imported previous executions. In this case the base name of the output files was set via the --cs-output switch. If omitted it will be deduced from the application name.

15.1.3  Result analysis

The results can be view interactively in the GUI provided by Coco. Run

coveragebrowser

to launch it. Select "File->Open…" to open the tcl.csmes file. You’ll see all Tcl files and procedures listed in the Sources and Methods tab, respectively. When selecting either of them a source code viewer will open that shows instrumented code with a gray background color. Note that the coloring is still limited to instrumented, i.e. seen code only. To check which code got actually executed (or not) select "File->Load Execution Report…" and open tcl.csexe. Now red and green colors denoting positive and negative coverage will show up for files, procedures and the code itself.

15.1.4  Filtered Tcl coverage run

As you might have noticed that above ad-hoc run produced results for several unnamed and internal files. These stem from internal Tcl/Tk files or e.g. mouse event handlers that result in on-the-fly eval calls. Compared to the application-specific Tcl files the coverage information for these internal pieces of code is of little interest.

The easiest way to hide the non-relevant pieces of code is through usage of the various --cs-exclude-code options. A recommended list of switches will be shown in the repetition of the previous run with filtering options applied:

coveragescannertcl --cs-output=tcl  --cs-exclude-code-regex="^\s*[^\n]*\s*$"
   --cs-exclude-code-regex="(tcl|tk)Init" --cs-exclude-code-regex="namespace eval ::itcl"
   --cs-exclude-code-regex="^# This file is generated"
   --cs-exclude-code-regex="tk::Menu(Motion|Leave)" ./proto

When viewing the new tcl.csmes in CoverageBrowser you’ll notice that the list of files is reduced to the relevant canvas.tclx, input.tclx, mainwindow.tclx, point.itcl and rect.itcl scripts.

15.1.5  Report Generation

All of text, XML and HTML reports can be generated from the CoverageBrowser GUI. With future automation of nightly test runs or Continuous Integration (CI) in mind we’ll create the report from the command line. And for this very purpose we’ll also pretend that the execution data has not been imported into the measurement database, yet. This step can be accomplished like this from the command line:

cmcsexeimport --title="First run" -m tcl.csmes -e tcl.csexe

The resulting proto.csmes file now contains everything we need to generated a report:

cmreport --title="Coverage Data" -m tcl.csmes --html=report.html

The resulting report.html file (including a report_html/ directory) can now be visited with a Web browser. It includes a listing of executions, source files, functions and metrics on a global level, per file and per functions.

15.2  Analysis of C/C++ code coverage

15.2.1  Instrumentation

While Tcl code can be instrumented for coverage analysis on-the-fly this step requires a rebuild for C or C++ applications. But this can be accomplished with almost no change to your regular application build. Here are the two possible approaches:

We’ll make use of approach number 2. First, clean the existing object files:

$ make clean

Then, recompile the application with a changed PATH and a switch that enables the compiler wrappers. For brevity, we’ll do this all in a single line:

$ PATH=~/SquishCoco/wrapper/bin:$PATH COVERAGESCANNER_ARGS="--cs-on"  make

As a result we’ll not only get the main executable (proto.exe) but also a corresponding instrumentation file proto.exe.csmes. Information about the instrumented code (gray background color) can optionally be reviewed at this point

coveragebrowser -m proto.exe.csmes

15.2.2  Execution

Let’s run the application

$ ./proto

and quit it via its GUI right after. The current working directory will now contain a proto.exe.csexe containing information about completed execution.

15.2.3  Result analysis

Same as with Tcl the results can now be reviewed in CoverageBrowser. A short cut invocation that opens both the instrumentation and execution files is shown here:

coveragebrowser -m proto.exe.csmes -e proto.exe.csexe

15.2.4  Report Generation

The process of a report generation works analogous to the approach shown for Tcl code above.

15.2.5  Analysis of mixed Tcl and C/C++ code coverage

Coverage data for the Tcl and C/C++ parts of a hybrid application can be gathered independently as shown above. Either in a single or multiple executions. The only extra step needed is the merging of the data into a single measurement database. Which then can be used to generate a single, combined report taking both languages into account.

15.2.6  Combined coverage run

Assuming the instrumentation of the C/C++ code is still in place we can gather coverage data for both types of languages with a single run:

coveragescannertcl --cs-output=tcl ./proto

With the proto.exe.csmes file still in place from the application rebuild we’ll now have two sets of files:

  1. tcl.csexe
  2. tcl.csmes
  3. proto.exe.csexe
  4. proto.exe.csmes

At this point the coverage data for Tcl and C/C++ could be analysed independently using cmcsexeimport. But we want to gain an overall picture and will therefore merge the results in the next section.

15.2.7  Merging measurement data

Instead of using the Coverage Browser GUI we’ll again use the command line tools. In the first step we’ll merge the instrumentation files stemming from the two languages into one:

cmmerge -o all.csmes tcl.csmes proto.exe.csmes

To save some disk space the --append option can be used rather than creating the new all.csmes file.

In the second step the two execution files are merged using the merge policy for the 2nd language:

cmcsexeimport -m all.csmes -e tcl.csexe --title="Hybrid" --passed
cmcsexeimport -m all.csmes -e proto.exe.csexe -p merge --title="Hybrid" --passed

For the merge to actually happen the title has to be identical.

15.2.8  Report Generation

An HTML report encompassing coverage for both application parts can be generated with a single command line call:

cmreport --title="Hybrid Tcl/C++ Coverage" -m all.csmes --html=report.html

Select "Source Files" to see both Tcl (various files) and C++ code (main.cpp) being listed in a combined report.

email squish@froglogic.com