Squish Coco

Code Coverage Measurement for Tcl, QML, C# and C/C++

Part VI
Tcl Coverage

Chapter 14  Command Line Arguments


coveragescannertcl [⟨cs-options⟩] ⟨application arguments


application arguments
Tcl-based application including optional arguments
Any of the following options:
Base name of the generated .csmes and .csexe files
Exclude files located in specified directory
Override exclusion of files based on specified directory
Exclude file if absolute file name matches regular expression
Override exclusion of file if absolute name matches regular expression
Exclude file if absolute name matches wildcard
Override exclusion of file if absolute name matches wildcard
Regular expression of code that will be instrumented even if it is excluded
Regular expression of code that will not be instrumented
Wildcard of code that will be instrumented even if it is excluded
Wildcard of code that will be instrumented even if it is excluded
Enables code instrumentation for standard Tcl initialization scripts
Instrument Tcl code even if it has no or just an ambiguous file name set. By default such code is ignored as it is typically just temporary code generated for e.g. an Tk event handler or results from nested eval calls.
Read command line options from the file at ⟨path⟩ and insert them at the position of this option. The option file is a text file with one option per line. Leading and trailing blanks and empty lines are ignored.

CoverageScannerTcl is a utility which collects code coverage information of a Tcl-based execution. It will collect all scripts parsed by the embedded Tcl interpreter and determine which parts were actually executed.

Once the application exits a pair of .csmes and .csexe files will be written out that contain the code instrumentation information and execution data, respectively.

If the --cs-output switch is omitted the base name of the output files will the same as the ⟨application⟩ name.

See (see Chapter 16) for a recommended set of --cs-exclude-code-regexp switches for typical Tcl/Tk applications.

Chapter 15  Getting started

At the moment, this example only works with Tcl v8.5 or earlier. See Section 15.1 below for details.

Create a file hello.tcl with the following content:

if { $argc == 0 } {
    puts "Hello World!"
else {
    for {set i 0} {$i<$argc} {incr i} {
        if { $i == 0 } {
            set hello "Hello [lindex $argv $i]"
        } elseif { $i == $argc - 1 } {
            set hello "$hello and [lindex $argv $i]"
        } else {
            set hello "$hello, [lindex $argv $i]"
    puts $hello

Execute hello.tcl using the tclsh interpreter:

coveragescannertcl tclsh hello.tcl Jim John James

Two files are generated:

The instrumentation database of hello.tcl
The execution report.

To import the execution report into its instrumentation database, execute:

cmcsexeimport -m tclsh.csmes --title="Jim John James" tclsh.csexe

After the import, it is possible to analyze the code coverage data directly with CoverageBrowser or generate a HTML report:

cmreport --title="Hello application" -m tclsh.csmes --html=tclsh.html

15.1  Using more than one Tcl version on one system

Squish Coco v3.0 does not yet support the use of tclsh under Tcl v8.6. If you want to use coveragescannertcl with tclsh, Tcl v8.5 or earlier is necessary.

In some UNIX®systems (like Debian Linux), more than one Tcl version can be present at the same time. The executables are then distinguished by their names. On such a system, the tclsh executable of Tcl v8.5 may be called tclsh8.5.

We can then run CoverageScannerTcl such that it calls tclsh8.5. On its own, this would however change the names of the generated files into tclsh8.5.csmes and tclsh8.5.csexe. We can correct this with another commandline option, --cs-output=tclsh. The first command in the example above now becomes:

coveragescannertcl --cs-output=tclsh tclsh8.5 hello.tcl Jim John James

Afterwards, the rest of the example can be done as before.

Chapter 16  Coco TCL Tutorial

16.1  Analysis of Tcl script coverage

16.1.1  Demo application build

Copy the sample to your working diretcory

cp -r /opt/SquishCoco/doc/demoapp_tcl ./
cd demoapp_tcl

Edit Makefile and adjust the following variables

TCLLIBS=-ltk8.5 -litcl3.4 -ltcl8.5

to match your system’s library names and include paths.
Build the C++ executable with:

$ make

Test the simple Tk GUI with an invocation of the launcher script:

$ ./proto

16.1.2  Ad-hoc Tcl coverage run

Launch the application under test via the Tcl coverage scanning tool :

coveragescannertcl --cs-output=tcl ./proto

The GUI will show up as normal while code coverage is monitored. At this point the application might already be shut down via the "File->Exit" menu entry or the [X] button in the window’s title bar. Observe the two files tcl.csmes and tcl.csexe being generated. The latter contains data about the code covered in the last execution. The former is a measurement database containing information about the instrumented code and possibly imported previous executions. In this case the base name of the output files was set via the --cs-output switch. If omitted it will be deduced from the application name.

16.1.3  Result analysis

The results can be view interactively in the GUI provided by Coco. Run


to launch it. Select "File->Open…" to open the tcl.csmes file. You’ll see all Tcl files and procedures listed in the Sources and Methods tab, respectively. When selecting either of them a source code viewer will open that shows instrumented code with a gray background color. Note that the coloring is still limited to instrumented, i.e. seen code only. To check which code got actually executed (or not) select "File->Load Execution Report…" and open tcl.csexe. Now red and green colors denoting positive and negative coverage will show up for files, procedures and the code itself.

16.1.4  Filtered Tcl coverage run

As you might have noticed that above ad-hoc run produced results for several unnamed and internal files. These stem from internal Tcl/Tk files or e.g. mouse event handlers that result in on-the-fly eval calls. Compared to the application-specific Tcl files the coverage information for these internal pieces of code is of little interest.

The easiest way to hide the non-relevant pieces of code is through usage of the various --cs-exclude-code options. A recommended list of switches will be shown in the repetition of the previous run with filtering options applied:

coveragescannertcl --cs-output=tcl  --cs-exclude-code-regex="^\s*[^\n]*\s*$"
   --cs-exclude-code-regex="(tcl|tk)Init" --cs-exclude-code-regex="namespace eval ::itcl"
   --cs-exclude-code-regex="^# This file is generated"
   --cs-exclude-code-regex="tk::Menu(Motion|Leave)" ./proto

When viewing the new tcl.csmes in CoverageBrowser you’ll notice that the list of files is reduced to the relevant canvas.tclx, input.tclx, mainwindow.tclx, point.itcl and rect.itcl scripts.

16.1.5  Report Generation

All of text, XML and HTML reports can be generated from the CoverageBrowser GUI. With future automation of nightly test runs or Continuous Integration (CI) in mind we’ll create the report from the command line. And for this very purpose we’ll also pretend that the execution data has not been imported into the measurement database, yet. This step can be accomplished like this from the command line:

cmcsexeimport --title="First run" -m tcl.csmes -e tcl.csexe

The resulting proto.csmes file now contains everything we need to generated a report:

cmreport --title="Coverage Data" -m tcl.csmes --html=report.html

The resulting report.html file (including a report_html/ directory) can now be visited with a Web browser. It includes a listing of executions, source files, functions and metrics on a global level, per file and per functions.

16.2  Analysis of C/C++ code coverage

16.2.1  Instrumentation

While Tcl code can be instrumented for coverage analysis on-the-fly this step requires a rebuild for C or C++ applications. But this can be accomplished with almost no change to your regular application build. Here are the two possible approaches:

We’ll make use of approach number 2. First, clean the existing object files:

$ make clean

Then, recompile the application with a changed PATH and a switch that enables the compiler wrappers. For brevity, we’ll do this all in a single line:

$ PATH=~/SquishCoco/wrapper/bin:$PATH COVERAGESCANNER_ARGS="--cs-on"  make

As a result we’ll not only get the main executable (proto.exe) but also a corresponding instrumentation file proto.exe.csmes. Information about the instrumented code (gray background color) can optionally be reviewed at this point

coveragebrowser -m proto.exe.csmes

16.2.2  Execution

Let’s run the application

$ ./proto

and quit it via its GUI right after. The current working directory will now contain a proto.exe.csexe containing information about completed execution.

16.2.3  Result analysis

Same as with Tcl the results can now be reviewed in CoverageBrowser. A short cut invocation that opens both the instrumentation and execution files is shown here:

coveragebrowser -m proto.exe.csmes -e proto.exe.csexe

16.2.4  Report Generation

The process of a report generation works analogous to the approach shown for Tcl code above.

16.2.5  Analysis of mixed Tcl and C/C++ code coverage

Coverage data for the Tcl and C/C++ parts of a hybrid application can be gathered independently as shown above. Either in a single or multiple executions. The only extra step needed is the merging of the data into a single measurement database. Which then can be used to generate a single, combined report taking both languages into account.

16.2.6  Combined coverage run

Assuming the instrumentation of the C/C++ code is still in place we can gather coverage data for both types of languages with a single run:

coveragescannertcl --cs-output=tcl ./proto

With the proto.exe.csmes file still in place from the application rebuild we’ll now have two sets of files:

  1. tcl.csexe
  2. tcl.csmes
  3. proto.exe.csexe
  4. proto.exe.csmes

At this point the coverage data for Tcl and C/C++ could be analysed independently using cmcsexeimport. But we want to gain an overall picture and will therefore merge the results in the next section.

16.2.7  Merging measurement data

Instead of using the Coverage Browser GUI we’ll again use the command line tools. In the first step we’ll merge the instrumentation files stemming from the two languages into one:

cmmerge -o all.csmes tcl.csmes proto.exe.csmes

To save some disk space the --append option can be used rather than creating the new all.csmes file.

In the second step the two execution files are merged using the merge policy for the 2nd language:

cmcsexeimport -m all.csmes -e tcl.csexe --title="Hybrid" --passed
cmcsexeimport -m all.csmes -e proto.exe.csexe -p merge --title="Hybrid" --passed

For the merge to actually happen the title has to be identical.

16.2.8  Report Generation

An HTML report encompassing coverage for both application parts can be generated with a single command line call:

cmreport --title="Hybrid Tcl/C++ Coverage" -m all.csmes --html=report.html

Select "Source Files" to see both Tcl (various files) and C++ code (main.cpp) being listed in a combined report.