Menu

Squish Coco

Code Coverage Measurement for Tcl, QML, C# and C/C++

Part X
Coco Integration Handbook

Chapter 41  Support for IDEs and toolkits

In this chapter we give examples how Squish Coco can be combined with several IDEs and build environments.

41.1  GNU Makefiles

Mostly, in makefiles, the C and C++ compiler and the linker are defined using the environment variables CC, CXX and LINK. This can be substituted by CoverageScanner by setting CC, CXX and LINK in the command arguments of make.

Examplemake LINK=csg++ CXX=csg++ CC=csgcc

41.2  CygWin

To install the CoverageScanner compiler wrapper for GCC and G++ on CygWin, proceed as follows:

  1. Open the Build Environment Selection application. (Windows Coco\toolselector.exe)
  2. Select the item “CygWin - www.cygwin.com”.
  3. Click on “Install CygWin Support”. The build environment selection dialog then displays the list of generated compiler wrappers.

Open then the CygWin console and compile your application with csgcc instead of gcc (or csg++ instead of g++).

41.3  Scratchbox

If Squish Coco is installed on the root file system1, a compiler wrapper is created for each compiler supported by Scratchbox. To invoke CoverageScanner, prepend cs to the name of the cross-compiler.

41.4  CMake

CMake is a platform independent build tool from Kitware which can be downloaded from http://www.cmake.org.

When Squish Coco is used with CMake, the changes are partially dependent on the tool chain that is used for compilation. We will now first describe the addition of a new build type, which is independent from the tool chain, and then the additional changes for Microsoft® Visual Studio® and GNU GCC.

41.4.1  Adding new build type for instrumented compilation

The first step is independent of the toolchain that is used. Its purpose is to declare the CMake variables that are used to specify the instrumented compilation. In CMake this is done by declaring a build type, which we will here call COVERAGE.

To do this, add to the to CMakeLists.txt file the following lines. (The variable COVERAGE_FLAGS in the first line specifies the CoverageScanner command line options. Change its value to fit you needs. Only --cs-on must always be present.)

SET(COVERAGE_FLAGS "--cs-on --cs-count")
SET(CMAKE_CXX_FLAGS_COVERAGE
    "${CMAKE_CXX_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the C++ compiler during coverage builds."
    FORCE )
SET(CMAKE_C_FLAGS_COVERAGE
    "${CMAKE_C_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the C compiler during coverage builds."
    FORCE )
SET(CMAKE_EXE_LINKER_FLAGS_COVERAGE
    "${CMAKE_EXE_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used for linking binaries during coverage builds."
    FORCE )
SET(CMAKE_SHARED_LINKER_FLAGS_COVERAGE
    "${CMAKE_SHARED_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the shared libraries linker during coverage builds."
    FORCE )
SET( CMAKE_STATIC_LINKER_FLAGS_COVERAGE
    "${CMAKE_STATIC_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the static libraries linker during coverage builds."
    FORCE )
MARK_AS_ADVANCED(
    CMAKE_CXX_FLAGS_COVERAGE
    CMAKE_C_FLAGS_COVERAGE
    CMAKE_EXE_LINKER_FLAGS_COVERAGE
    CMAKE_SHARED_LINKER_FLAGS_COVERAGE
    CMAKE_STATIC_LINKER_FLAGS_COVERAGE
    COMPILE_DEFINITIONS_COVERAGE
)

These commands take the compiler and linker flags of the “Release” build type and add to them the coverage flags. If you want to use instead the flags of another build type, replace the suffix “_RELEASE” in this code with the name of another build type, e.g. with “_DEBUG”.

41.4.2  Microsoft® Visual Studio®

Under Microsoft® Visual Studio®, we need to make the new Coverage build type visible to the IDE. To do this, add to CMakeLists.txt the lines

if(CMAKE_CONFIGURATION_TYPES)
   set(CMAKE_CONFIGURATION_TYPES Debug Release MinSizeRel RelWithDebInfo Coverage)
   set(CMAKE_CONFIGURATION_TYPES "${CMAKE_CONFIGURATION_TYPES}" CACHE STRING
     "Reset the configurations to what we need" FORCE)
endif()

If necessary, customize the list of configuration types according to your needs. To make the changes visible to Microsoft® Visual Studio®, a complex procedure is apparently needed:2

  1. Compile CMakeLists.txt with Ctrl-F7. If warnings occur, they can be ignored.
  2. Make a trivial change in CMakeLists.txt, like adding a space.
  3. Compile again. Then a dialog, "File modification detected", should appear. Click "Reload".

Then the list of solution configurations is updated and one can compile in coverage mode.

41.4.3  Compilation with Microsoft® NMake

In a project that is compiled whith NMake, the following must be done.

  1. Create a toolchain definition file cl.cmake which replaces the compiler and linker with their CoverageScanner wrappers.

    Example

    # this one is important
    SET(CMAKE_SYSTEM_NAME Windows)

    # specify the cross compiler
    FILE(TO_CMAKE_PATH "$ENV{SQUISHCOCO}/visualstudio" SQUISHCOCO)
    SET(CMAKE_C_COMPILER ${SQUISHCOCO}/cl.exe
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_CXX_COMPILER ${SQUISHCOCO}/cl.exe
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_LINKER ${SQUISHCOCO}/link.exe
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
  2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE.

    Example

    cmake.exe -DCMAKE_TOOLCHAIN_FILE=cl.cmake -DCMAKE_BUILD_TYPE=COVERAGE \
              -G "NMake Makefiles" ⟨path of cmake project
  3. Build the project with nmake.

41.4.4  Compilation with GNU GCC

The following must be done in a project that is comiled with gcc.

  1. Create a toolchain definition file gcc.cmake which replaces the compiler and linker with their CoverageScanner wrappers.

    Example

    find_program(CODE_COVERAGE_GCC gcc
        PATHS /opt/SquishCoco/wrapper/bin "$ENV{HOME}/SquishCoco/wrapper/bin"
        NO_DEFAULT_PATH )
    find_program(CODE_COVERAGE_GXX g++
        PATHS /opt/SquishCoco/wrapper/bin "$ENV{HOME}/SquishCoco/wrapper/bin"
        NO_DEFAULT_PATH )
    find_program(CODE_COVERAGE_AR ar
        PATHS /opt/SquishCoco/wrapper/bin "$ENV{HOME}/SquishCoco/wrapper/bin"
        NO_DEFAULT_PATH )

    # specify the cross compiler
    SET(CMAKE_C_COMPILER "${CODE_COVERAGE_GCC}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_CXX_COMPILER "${CODE_COVERAGE_GXX}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_LINKER "${CODE_COVERAGE_GXX}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_AR "${CODE_COVERAGE_AR}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
  2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE.

    Example

    cmake -DCMAKE_TOOLCHAIN_FILE=gcc.cmake -DCMAKE_BUILD_TYPE=COVERAGE \
          -G "Unix Makefiles" ⟨path of cmake project
  3. Build the project with make.

41.5  Qt Framework

41.5.1  Qt Creator

The following steps are needed to add instrumentation to a Qt Creator project.

With these settings, a new build mode CodeCoverage has been created that creates an instrumented version of your program.

If the project has unit tests, it is possible to execute them at every build and also generate a coverage report for them. In order to do this, the following additional steps are needed.

Now, every time you click on Build (not just Run) with this configuration, the project will be built, instrumented and optionally a report will be generated. Note that in case the project has been built before this configuration was performed, it will be required to execute the Clean Build step before building, so that the coverage measurement data will be generated.

Also keep in mind that when there was an error and one of the build steps exits with a non-zero exit status code, Qt Creator stops the build and displays the error message of the executable that failed, instead of continuing the build. This means that if a unit test fails, instead of cmcsexeimport running and importing the execution as a failed execution, cmcsexeimport does not run at all until the test case failure was fixed.

41.5.2  qmake

qmake is a tool to generate a Makefile in a platform-independent way, using as specification a so-called “project file”. By default, qmake chooses the programs that are used for compilation. We can however use the Squish Coco wrappers by setting some of qmake’s variables to new values. This can be done via command line parameters or by editing the project configuration files.

Command line.

In simple cases, we can do this on the command line. There are two methods.

Project configuration.

In more complex cases it is better to enable coverage by putting some declarations in the qmake project files. Since one needs to build the project with and without coverage, the new definitions must be put into a scope. (A scope is a region in a qmake project file that can be activated on demand.)

The following listing shows a template for such a scope, named CoverageScanner, which should be sufficient for most most projects.

CodeCoverage {
  COVERAGE_OPTIONS =

  QMAKE_CFLAGS   += $$COVERAGE_OPTIONS
  QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS
  QMAKE_LFLAGS   += $$COVERAGE_OPTIONS

  QMAKE_CC=cs$$QMAKE_CC
  QMAKE_CXX=cs$$QMAKE_CXX
  QMAKE_LINK=cs$$QMAKE_LINK
  QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB
  QMAKE_AR=cs$$QMAKE_AR
  QMAKE_LIB=cs$$QMAKE_LIB
}

Here we also set the variables QMAKE_LINK_SHLIB and QMAKE_AR, which contain the names of the command to link shared libraries and that to generate archives. Furthermore, can use COVERAGE_OPTIONS to set coveragescanner commandline options (see Chapter 9.2) to customize the project. An empty value for COVERAGE_OPTIONS will also work and results in a default instrumentation.

In a small project, the CodeCoverage scope may then be copied to all profile files of the project, i.e. those that end in “.pro”. (In fact, it is enough to insert them only into those files that actually compile code and not those that only include other files.) If the project is larger, it has very often a file with common settings that is included by all profiles: This is then the most convenient place to insert the CodeCoverage scope only once.

The new coverage scope is by default inactive. To enable code coverage in a project you want to built, just add the name of the scope to the CONFIG variable when configuring it with qmake:

$ qmake CONFIG+=CodeCoverage

41.5.3  moc

The Meta-Object Compiler moc adds automatically new methods to each classes derived from QObject. For example, the translation function tr, the source code for all signals, the cast operator qt_cast, …In order to instrument the code using the Qt Framework and not the Qt code itself, CoverageScanner provides the command line options --cs-qt3 for Qt3 and --cs-qt4 for Qt4 and Qt5. They are enabled by default.

In this case:

41.5.4  qbs

To use CoverageScanner with the Qt Build Suite (qbs) it can be set up as a toolchain. This toolchain can then be used for all qbs projects.

In order to set up CoverageScanner as a toolchain, issue the following command:

qbs setup-toolchains --type gcc /opt/SquishCoco/bin/csgcc csgcc

For Unix-based operating systems some additional configuration steps are necessary:

qbs config profiles.csgcc.cpp.archiverPath /opt/SquishCoco/bin/csar
qbs config profiles.csgcc.cpp.linkerName csg++
qbs config profiles.csgcc.cpp.nmPath /opt/SquishCoco/bin/csgcc-nm

The csgcc toolchain can then also be used as base profile for Qt projects:

qbs setup-qt /opt/Qt/bin/qmake qt-csgcc
qbs config profiles.qt-csgcc.baseProfile csgcc

41.6  SCons

To use Squish Coco with SCons, proceed as followings:

Here is a code snippet which can be used for Microsoft® Visual Studio® command line tools:

import os
from os.path import pathsep

env = Environment()

# Add the path of Squish Coco compiler wrapper
env[ 'ENV' ][ 'PATH' ] = os.environ[ 'SQUISHCOCO' ] + pathsep + env[ 'ENV' ][ 'PATH' ]
# TEMP variable need to be defined
env[ 'ENV' ][ 'TEMP' ] = os.environ[ 'TEMP' ]
env[ 'ENV' ][ 'INCLUDE' ] = os.environ[ 'INCLUDE' ]

# Set the compiler to Squish Coco wrappers
env[ 'CC' ]   = 'cs' + env[ 'CC' ] ;
env[ 'AR' ]   = 'cs' + env[ 'AR' ] ;
env[ 'LINK' ] = 'cs' + env[ 'LINK' ] ;

# Code coverage settings
coverageflags = [ '--cs-count' ]
env[ 'CCFLAGS' ]   = env[ 'CCFLAGS' ] + coverageflags ;
env[ 'ARFLAGS' ]   = env[ 'ARFLAGS' ] + coverageflags ;
env[ 'LINKFLAGS' ] = env[ 'LINKFLAGS' ] + coverageflags ;

In order to set correctly the build environment, it is necessary to start the build from Visual Studio’s developper prompt (is available through the start menu) or Squish Coco’s console provided by the "Build Environment Seletction".

41.7  ARM® Keil® µVision

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper for ARM and enable the code coverage analysis during the compilation.

Processed as followings:

  1. To install CoverageScanner compiler wrapper for the ARM® Keil® µVision tool chain, proceed as follows:
    1. Open the Build Environment Selection application. (Windows Coco\toolselector.exe)
    2. Select the item “ARM-Keil”.
    3. Select the directory o which the ARM® Keil® µVision compilers are installed (ex: C:\Keil_v5\ARM)
    4. Click on “Install ARM-Keil Support” and wait for the confirmation dialog that the tools are generated.
  2. Activate the code coverage analysis during the compilation
    1. Click on "Project->Option for Target…"
    2. On the "C/C++" tab, add --cs-on into the "Misc Controls" field.
      pictures/manual.tmp041.png
      Figure 41.1: Activating the code coverage analysis for an ARM® compiler
    3. On the "Linker" tab, add --cs-on into the "Misc Controls" field.
      pictures/manual.tmp042.png
      Figure 41.2: Activating the code coverage analysis for an ARM® linker

41.8  Arm® DS

To enable the code coverage analysis for an Arm® DS project, it is first necessary to compile it with Squish Coco compiler wrapper for ARM and enable the code coverage analysis during the compilation.

Processed as followings:

  1. To install CoverageScanner compiler wrapper for the Arm® DS tool chain, proceed as follows:
    1. Open the Build Environment Selection application. (Windows Coco\toolselector.exe)
    2. Select the item “ARM-DS”.
      pictures/manual.tmp043.png
      Figure 41.3: Installation of Arm® DS support in Build Environment Selection.
    3. Select the directory o which the Arm® DS compilers are installed (ex: C:\Program Files\DS-5 v5.29.3)
    4. Click on “Install ARM-DS Support” and wait for the confirmation dialog that the tools are generated.
      pictures/manual.tmp044.png
      Figure 41.4: Comfirmation of the installation of Arm® DS support.
  2. Activate the code coverage analysis during the compilation
    1. Click on "Project->Properties"
    2. On the "Arm C/C++ Compiler 5" tab, add --cs-on into the "Miscellaneous" field.
      pictures/manual.tmp045.png
      Figure 41.5: Activating the code coverage analysis for an ARM® compiler
    3. On the "Arm Linker 5" tab, add --cs-on into the "Misc Controls" field.
      pictures/manual.tmp046.png
      Figure 41.6: Activating the code coverage analysis for an ARM® linker

In order to upload the coverage data from the target to the host, it is necessary to define some custom I/O functions and trigger the coverage generation by calling __coveragescanner_save(). Due to the fact that Arm® DS supports semi-hostings extensions, the simplest way is to use it to generate the execution report directly on the host file system.

For that modify the main() function to add a call to __coveragescanner_set_custom_io(), __coveragescanner_filename() and __coveragescanner_save() as in the following sample:

Syntax:

#include <stdio.h>


#ifdef __COVERAGESCANNER__
static int csfputs(const char *s, void *stream)
{
  return fputs(s, (FILE *)stream);
}

static void *csfopenappend(const char *path)
{
  return (void*)fopen(path,"a+");
}


static int csfclose(void *fp)
{
  return fclose((FILE*)fp);
}
#endif


int
 main() {
#ifdef __COVERAGESCANNER__
  __coveragescanner_set_custom_io( NULL,
      csfputs,
      csfopenappend,
      NULL,
      NULL,
      csfclose,
      NULL);
  __coveragescanner_filename( "c:\\tmp\\test_arm5_csexe" ); /* destination file on the host file system */
#endif
  .......
#ifdef __COVERAGESCANNER__
    __coveragescanner_save(); /* saves the execution report */
#endif
        return 0;
}

The project can now be rebuilt and a project file with the extension .afx.csmes will the generated. This file can be opened in CoverageBrowser. Execute then the target application and wait until __coveragescanner_save() is called. A file c:\tmp\test_arm5_csexe will be generated (hard coded in our sample). Import it using CoverageBrowserand the coverage can be analyzed as in the following screenshot:

pictures/manual.tmp047.png
Figure 41.7: Code coverage of an Arm® DS project
The extension of an execution report is usually .csexe but the semi-hosting implementation provided by ARM remove the file extensions. It is necessary to remove the default file filter in the import dialog of CoverageBrowser to be able to import it.

41.9  Green Hills® Software MULTI Launcher

41.9.1  Installation

To install CoverageScanner compiler wrapper for the Green Hills® Software tool chain, proceed as follows:

  1. Open the Build Environment Selection application. (Windows Coco\toolselector.exe)
  2. Select the item “Green Hills”.
  3. Select the directory o which the Green Hills® Software compilers are installed
  4. Click on “Install Green Hills Support” and wait for the confirmation dialog that the tools are generated.

41.9.2  Command Line Tools

The command line tools for the Green Hills® Software compiler are installed in the folder squishcoco of the installed native tool chain. Example: if the native tool chain is installed under c:\ghs\comp_201426, the Squish Coco compiler wrapper are installed under c:\ghs\comp_201426\squishcoco.

The compiler wrappers are replacing completely the Green Hills® Software tool chain. To activate the code coverage analysis, it is necessary to add the parameter --cs-on to the compiler arguments.

41.9.3  CoverageScanner Library

In order to save an execution report, it is necessary to:

  1. provide to the CoverageScanner library a list of I/O functions which let you upload the coverage information to the host.
  2. Save manually the coverage report on a specific trigger.

To save a coverage report the following code snippet can be used (an this code in an event handler which should trigger the report generation):

   1 #ifdef __COVERAGESCANNER__
   2     __coveragescanner_save();
   3     __coveragescanner_clear();
   4 #endif

To provide the i/O functions it is necessary to call __coveragescanner_set_custom_io() in the main() function of the application. At least three functions need to be provided by __coveragescanner_set_custom_io():

  1. An equivalent function of fopen() which opens the execution report file (with the extension .csexe) in append mode.
  2. An equivalent function of fclose()
  3. An equivalent function of fputs() to transfer the contain.

Example:  The following code write an execution report to the local file system using the C file API.

   1 #ifdef __COVERAGESCANNER__
   2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }
   3 static void *csfopenappend(const char *path)    { return (void*)fopen(path,"a+");  }
   4 static int csfclose(void *fp)                   { return fclose((FILE*)fp);        }
   5 #endif
   6 
   7 int main()
   8 {
   9 #ifdef __COVERAGESCANNER__
  10     __coveragescanner_set_custom_io( NULL,
  11             csfputs,
  12             csfopenappend,
  13             NULL,
  14             NULL,
  15             csfclose,
  16             NULL);
  17 #endif
  18 ....
  19 }

41.10  VisualDSP®

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper and enable the code coverage analysis during the compilation in the VisualDSP® configuration.

Processed as followings:

  1. Click on "Project->Project Options…"
  2. On the "Compile" tab, add --cs-on into the "Additional options" field.
    pictures/manual.tmp048.png
    Figure 41.8: Activating the code coverage analysis for the compiler
  3. On the "Link" tab, add --cs-on into the "Additional options" field.
    pictures/manual.tmp049.png
    Figure 41.9: Activating the code coverage analysis for the linker
  4. Rebuild your project.

On most embedded targets, it is necessary to provide a dedicated I/O since no file systems are available for storing the code coverage information. Also since embedded applications generally never exit, it is often necessary to implement an event handler which saves the execution report upon the reception of a specific trigger.

The simulator emulates the support of a file system. To save the coverage report on the current build directory, it is only necessary to register in the first lines of the main() a custom file I/O which uses the standard C file API.

Example

   1 #ifdef __COVERAGESCANNER__
   2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }
   3 static void *csfopenappend(const char *path)    { return (void*)fopen(path,"a+");  }
   4 static int csfclose(void *fp)                   { return fclose((FILE*)fp);        }
   5 #endif
   6 
   7 int main()
   8 {
   9 #ifdef __COVERAGESCANNER__
  10     __coveragescanner_set_custom_io( NULL,
  11             csfputs,
  12             csfopenappend,
  13             NULL,
  14             NULL,
  15             csfclose,
  16             NULL);
  17 #endif
  18 ....
  19 }

To record a coverage report when a specific trigger occurs, add the following source code lines in its handler:

   1 #ifdef __COVERAGESCANNER__
   2     __coveragescanner_save();
   3     __coveragescanner_clear();
   4 #endif

41.11  Microsoft® Visual Studio®

Squish Coco provides a wrapper for link.exe and cl.exe located on the %SQUISHCOCO%\visualstudio directory. It behaves exactly like the corresponding Microsoft® wrapper except that the code coverage analysis becomes activated when the option --cs-on is added to the command arguments. These wrappers call the Microsoft® tools for compilation or for linkage.

41.11.1  Microsoft® Visual Studio® .NET C# Compiler

To activate the instrumentation of C# source code, it is only necessary to add the symbol COVERAGESCANNER_COVERAGE_ON in the properties of the Microsoft® Visual Studio® .NET project. Other symbols can be appended to select additional instrumentation options. The full list can be found on the chapter 9.3.

41.11.2  Microsoft® Visual Studio® .NET C and C++ Compiler

To use Squish Coco with Microsoft® Visual Studio® .NET proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.
    For Microsoft® Visual Studio® 2005 or 2008:
    1. Start Microsoft® Visual Studio® 2005 or 2008.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Projects->VC++ Directories".
    4. Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 41.10)
      pictures/manual.tmp050.png
      Figure 41.10: Installation of CoverageScanner on Microsoft® Visual Studio® 2005 and 2008: Setting the path of CoverageScanner.

    For Microsoft® Visual Studio® 2010:

    1. Start Microsoft® Visual Studio® 2010.
    2. Open a C++ project.
    3. Open the project properties using the context menu of the laded project.
    4. Select the item "Configuration Properties->VC++ Directories".
    5. Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of executable directories. (see Figure 41.11)
      pictures/manual.tmp051.png
      Figure 41.11: Installation of CoverageScanner on Microsoft® Visual Studio® 2010: Setting the path of CoverageScanner.
  2. To activate code coverage analysis:
    1. Open a Visual C or C++ project.
    2. Edit the project settings (click on "Project->Properties").
    3. Add to the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 41.12)
    4. In the additional arguments of the linker, add the --cs-libgen which specifies which library should be used for the generation of the CoverageScanner library. The table 41.1 contains the list of recommended settings.
    5. For Microsoft® Windows CE applications, append to the linker arguments the command line option --cs-architecture which lets you specify the target platform. The table 41.2 contains the list of available architectures.
LibraryLibrary FileCommand line option
Single ThreadedLIBC.LIB--cs-libgen=/ML
Static MultiThreadLIBCMT.LIB--cs-libgen=/MT
Dynamic Link (DLL)LIBCRT.LIB--cs-libgen=/MD
Debug Single ThreadedLIBCD.LIB--cs-libgen=/MLd
Debug Static MultiThreadLIBCMTD.LIB--cs-libgen=/MTd
Debug Dynamic Link (DLL)LIBCRTD.LIB--cs-libgen=/MDd
Table 41.1: CoverageScanner library settings for Microsoft® Visual Studio®
Targeted ArchitectureCommand line option
ARM Microprocessor--cs-architecture=ARM
ARM Microprocessor (Thumb code)--cs-architecture=THUMB
x86 Microprocessor--cs-architecture=IX86
MIPS16 Microprocessor--cs-architecture=MIPS16
MIPS Microprocessor--cs-architecture=MIPS
MIPS Microprocessor with FPU--cs-architecture=MIPSFPU
SH3 Microprocessor with FPU--cs-architecture=SH3
SH4 Microprocessor with FPU--cs-architecture=SH4
Table 41.2: List of target architectures
pictures/manual.tmp052.png
Figure 41.12: Activation of the instrumentation under Visual Studio® .NET.
pictures/manual.tmp053.png
Figure 41.13: Activation of the instrumentation under Visual Studio® .NET.

41.11.3  Microsoft® Visual C++ Express

To use Squish Coco with Microsoft® Visual C++ Express proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.
    1. Start Microsoft® Visual C++ Express.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Projects->VC++ Directories".
    4. Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 41.14)
      pictures/manual.tmp054.png
      Figure 41.14: Installation of CoverageScanner on Visual C++ Express: Setting the path of CoverageScanner.
  2. The activation of the code coverage analysis is similar to Microsoft® Visual Studio® .NET. (see Chapter 41.11.2)

41.11.4  Microsoft® Visual Studio® 6.0

To use Squish Coco with Microsoft® Visual Studio® 6.0 proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.
    1. Start Microsoft® Visual Studio® 6.0.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Directories".
    4. Select "Executable files" in the combobox "Show directories for:".
    5. Add the path of the directory visualstudio of the Squish Coco installation3 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\visualstudio, (see Figure 41.15))
      pictures/manual.tmp055.png
      Figure 41.15: Installation of CoverageScanner on Visual Studio® 6.0: Setting the path of CoverageScanner.
  2. To activate the code coverage analysis:
    1. Open a Visual C or C++ project.
    2. Edit the project settings (click on "Project->Properties").
    3. Add the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 41.16)
    4. In the additional arguments of the linker, add the --cs-libgen which specifies the name of the generated CoverageScanner library. The table 41.1 contains the list of recommended settings.
pictures/manual.tmp056.png
Figure 41.16: Activation of the instrumentation under Visual Studio® 6.0.
pictures/manual.tmp057.png
Figure 41.17: Activation of the instrumentation under Visual Studio® 6.0.

41.12  Microsoft® eMbedded Visual C++®

To use Squish Coco with Microsoft® eMbedded Visual C++®proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.
    1. Start Microsoft® eMbedded Visual C++®.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Directories".
    4. Select "Executable files" in the combobox "Show directories for:".
    5. Select "Platform" and the targeted "CPUs".
    6. Add the path of the directory WinCE of the Squish Coco installation4 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\WinCE, (see Figure 41.18))
      pictures/manual.tmp058.png
      Figure 41.18: Installation of CoverageScanner on eMbedded Visual C++®: Setting the path of CoverageScanner.
  2. To activate the code coverage analysis:
    1. Open a Visual C or C++ project.
    2. Edit the project settings (click on "Project->Properties").
    3. Add the option --cs-on to the additional command line arguments of the C and C++ compiler and linker. (see Figure 41.19)
pictures/manual.tmp059.png
Figure 41.19: Activation of the instrumentation under eMbedded Visual C++®.
pictures/manual.tmp060.png
Figure 41.20: Activation of the instrumentation under eMbedded Visual C++®.

41.13  Eclipse IDE for C/C++

In Eclipse IDE for C/C++, code coverage is enabled for a configuration by replacing the names of the compilers it uses with the names of the Squish Coco compiler wrappers. This can be done in the following way:

  1. Start Eclipse.
  2. Load the C or C++ project that should be instrumented.
  3. Open the property window (Project-> Properties).
  4. Click on "C/C++ Build/Settings".
  5. Create a new configuration by clicking on "Manage Configurations...", and select it.
  6. Click on "Tools Settings" tab.
  7. Click on "GCC C++ Compiler" and prepend cs to the name of the compiler.
  8. Click on "GCC C Compiler" and prepend cs to the name of the compiler.
  9. Click on "C++ Linker" and prepend cs to the name of the linker.
  10. If it is a library project, also click on "GCC archiver" and replace the name of the archiver command ar with csar.

Now the Eclipse will use the Squish Coco wrappers instead of the compilers when compiling.

pictures/manual.tmp061.png
Figure 41.21: Eclipse settings

However, the Squish Coco wrappers are not by default in the search part and can not yet be found during compilation.

To change this, a copy of the PATH variable needs to be added to the "C/C++ Build->Environment" section of the property window, and the path of the Squish Coco binaries added. Under UNIX®, and with Squish Coco installed at the default location, PATH will then have a value like “/opt/SquishCoco/bin:/usr/local/bin:/usr/bin:/bin”.

Almost always it will be necessary to modify the behavior of Squish Coco by setting command line options. The easiest way to do this is by adding the variable COVERAGESCANNER_ARGS in "C/C++ Build->Environment". Its value then consists of command line options (see Chapter 9.2). Code instrumentation is however already activated if this variable is not set or empty.

If the value of COVERAGESCANNER_ARGS has changed, it is necessary to compile the whole project again; otherwise the new options have no effect.

If everything is done correctly and the project is compiled, one will see in the console window that csg++ is used instead of g++, etc., and that a .csmes file is created next to the place of the newly-built binary. When the binary then is run, a .csexe file is created in its working directory, which is typically a different directory from that of the .csmes file.

41.14  Apple® Xcode

To use Squish Coco with Apple® Xcode proceed as follows: To activate the code coverage analysis:

  1. Open a terminal window and set the CPLUSPLUS, LDPLUSPLUS, LD and CC to CoverageScanner compiler wrapper. The path of native compiler (clang, clang++, gcc or g++) need to be present in the PATH environment variable. Start Xcode using the open command.

    If GCC is used as compiler:

    SQUISHCOCO=/Applications/SquishCoco/wrapper
    export CC=$SQUISHCOCO/gcc
    export LD=$SQUISHCOCO/gcc
    export CPLUSPLUS=$SQUISHCOCO/g++
    export LDPLUSPLUS=$SQUISHCOCO/g++

    open /Developer/Applications/Xcode.app

    If clang is used as compiler:

    SQUISHCOCO=/Applications/SquishCoco/wrapper
    export CC=$SQUISHCOCO/clang
    export LD=$SQUISHCOCO/clang
    export CPLUSPLUS=$SQUISHCOCO/clang++
    export LDPLUSPLUS=$SQUISHCOCO/clang++
    export XCODE_TOOLCHAIN_DIR=/Applications/Xcode.app/Contents/Developer/Toolchains
    export XCODE_TOOLCHAIN=$XCODE_TOOLCHAIN_DIR/XcodeDefault.xctoolchain/usr/bin/

    export PATH=$XCODE_TOOLCHAIN:$PATH

    open /Applications/Xcode.app
  2. Open a Xcode C or C++ project.
  3. Edit the project settings (click on "Project->Edit Project Settings").
  4. Add the option --cs-on to the additional command line arguments of the C and C++ compiler (fields Other C Flags and Other C++ Flags) and linker (field Other Linker Flags). (see Figure 41.22)
  5. Disable the usage of the precomiled header: Open the settings of the active target ("Project->Edit Active Target") and remove the contains of Prefix Header.
pictures/manual.tmp062.png
Figure 41.22: Activation of the instrumentation under Apple® Xcode.

41.14.1  VxWorks support on Linux

To activate Squish Coco on Wind River’s Workstation proceed as follows:

  1. Start Wind River’s Workbench
  2. Select your project in the "Project Explorer" window and click on the entry "Properties" in the context menu.
  3. Select the build properties, the compiler tool chain and click on the "Variables" tab.
  4. Search for the TOOL_PATH variable and replace the /bin string at the end with /squishcoco:
    pictures/manual.tmp063.png
    Figure 41.23: Setting the TOOL_PATH variable

After this change, the project can be compiled with CoverageScanner or the native tool chain. To use CoverageScanner, it is necessary to add the option --cs-on to the command line arguments of the compiler and the linker.

To do this, proceed as followings:

  1. Start the Wind River Workbench
  2. Select your project in the "Project Explorer" window and click on "Properties" of the context menu.
  3. Select the build properties, the compiler tool chain and click on the "Tools" tab.
  4. Select the entry "C-Compiler" in the "Build Tool" combo box and add to the content of the "Tool Flags..." field the argument --cs-on:
    pictures/manual.tmp064.png
    Figure 41.24: Activating the code coverage analysis
  5. Do the same for the “C++-Compiler” and the “Linker” tool.

With these settings, a file with the extension .vxe.csmes is generated when the code is compiled. It contains the complete instrumented code and can be inspected with CoverageBrowser. The resulting target application will then create a file with the suffix .csexe after it has run.

This file is created on the target file system. It can them be transferred to the host and imported into the .vxe.csmes file to create a report.



A video that illustrates the content of this chapter is available on https://youtube.com/watch?v=#1.

Code Coverage with Squish Coco of a VxWorks application on Linux

 

41.15  IAR Embedded Workbench

Only Windows OS supported. The IAR Embedded Workbench should be already installed. Execute the "Build Environment Selection" program after the installation of Coco. The "Build Environment Selection" program can be found in the Coco install folder with the name "toolselector.exe". In the main window select the correct installation folder for your IAR binary folder and click Install IAR Support

pictures/manual.tmp065.png
Figure 41.25: Build Environment Selection

The next step is to adjust your Embedded Workbench to execute Coco. To this by clicking Extra Options in the project settings, found in: Project > Options > C/C++ Compiler > Extra Options

pictures/manual.tmp066.png
Figure 41.26: Compiler Extra Options

Enable the checkbox and type “--cs-on” into the field. Also add this to the linker by clicking the following in the menu. Project > Options > Linker > Extra Options

pictures/manual.tmp067.png
Figure 41.27: Linker Extra Options

Enable the checkbox and type “--cs-on” into the field. 
To deactivate Coco just uncheck the two checkboxes again.

41.16  TASKING Support

The "TriCore Eclipse IDE" from TASKING should already be installed. Execute the "Build Environment Selection" program after the installation of Coco. The "Build Environment Selection" program can be found in the Coco install folder with the name "toolselector.exe". In the main window select "TASKING VX-Toolset for TriCore" and click on "Install TASKING Support"

pictures/manual.tmp068.png
Figure 41.28: Build Environment Selection

The next step is to activate the code coverage support in the Eclipse IDE. Open "TriCore Eclipse IDE", click on the top of your project and with the contect menu, select "Properties".

The project’s configuration dialog will be opened and then select "C/C++ Build->Setting" on the side bar. The complete compiler and linker settings will be displayed. Select then "C/C++ Compiler->Miscellaneous". You shoudl see then an additional argument as in the following screenshot:

pictures/manual.tmp069.png
Figure 41.29: Compiler Extra Options

Add “--cs-on” to enable the code coverage analysis for the compiler.

Then do the same for the linker, select "Linker->Miscellaneous" and add “--cs-on” to the additional arguments.

pictures/manual.tmp070.png
Figure 41.30: Linker Extra Options

It is then possible to instrument the embedded application by rebuilding it. A .csmes file is then generated upon the compilation.

But to see the coverage it is necessary to provide a set of functions that permist to upload it to the host. The TASKING simulator support ARM’s semi-hosting function which permits to write on file directly from the target to the host’S file system. We can use this to generate the execution report (.csexe file) but on real hardware, it may be necessary to use other implementations.

For teh setup, add the following lines to the source containing the main function:

#include <stdio.h>

#ifdef __COVERAGESCANNER__
static int csfputs(const char *s, void *stream)
{
  return fputs(s, (FILE *)stream);
}

static void *csfopenappend(const char *path)
{
  return (void*)fopen(path,"a+");
}

static int csfclose(void *fp)
{
  return fclose((FILE*)fp);
}
#endif

Then in the first lines of the main() function, configure these functions as handler:

int main( int argc, char *argv[] )
{
#ifdef __COVERAGESCANNER__
    __coveragescanner_set_custom_io(
        NULL,
        csfputs,
        csfopenappend,
        NULL,
        NULL,
        csfclose,
        NULL
        );
#endif

    ......
}

Then at the location where the execution report should be saved, add the following lines:

#ifdef __COVERAGESCANNER__
    __coveragescanner_save();
#endif

Chapter 42  Support for specific test frameworks

42.1  CppUnit

CppUnit1 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

The following code is an example how this can be done:

To download: CppUnitListener.cpp pictures/zoom.png

In the example, we have done the following steps:

  1. We write a CppUnit listener class which records the code coverage of unit each test after it is completed.

    We want to be able to run the program with and without Squish Coco. Therefore we use in the code the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Squish Coco, without the need to #include anything.

    In the listener class, CppUnitListener, we use the following menber functions:

    startTest()
    This function is called before each test begins.

    In it, we compute a test name with the information provided by CppUnit and pass it to the Squish Coco library with __coveragescanner_testname().

    We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored.

    addFailure()
    This function is called after a test fails. It just sets a flag that is used by the other functions.
    endTest()
    This function is called after a test has ended.

    It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  2. We call __coveragescanner_install() in the main() function.
  3. We add this listener in the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines:
    CoverageScannerListener coveragescannerlistener;
    controller.addListener( &coveragescannerlistener );

42.2  QTestLib

QTestLib is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test.

Proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end of every unit test.
  3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject.
  4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them to initTest() and cleanupTest().
  5. Compile your project with code code coverage enabled.

TestCoverageObject header:

To download: testcoverageobject.h pictures/zoom.png

TestCoverageObject source:

To download: testcoverageobject.cpp pictures/zoom.png

42.3  GoogleTest

GoogleTest2 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

Simply proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a TestEventListener class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener give an implementation example.
  3. Add this listener in the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()).
  4. Compile the unit test using CoverageScanner.
To download: googletest.cpp pictures/zoom.png

42.4  CxxTest

CxxTest3 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

Proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener.

    In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test completion. To ensure to get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()).

    When an test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save().

    Finally, all test failure members of this class must be reimplemented to record the test failures.

  3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter().run().
  4. Compile the unit test with CoverageScanner activated.

Example

To download: CxxTestListener.cpp pictures/zoom.png

42.5  boost::test

boost::test4 is a unit test framework for C++ which is part of the boost libraries.

Proceed as follows:

  1. Implement a TestObserver, which derives from 'boost::unit_test_framework::test_observer' and store as "BoostTestObserver.hpp".
  2. Also implement (within the same file) a ’boost::test::fixture’ and define it to be executed in the beginning and in the end of each executed boost::test by: BOOST_GLOBAL_FIXTURE(FixtureName). The file then should look something similar to this:
    To download: BoostTestObserver.hpp pictures/zoom.png

  3. If it is included by any of your allready existing ’boost::test’ unit test .cpp files (must be done right after the in ’boost::test’ mandatory  BOOST_TEST_MODULE definition), It is being instantciated and initialized by the boost::test automatically.
  4. The ’boost::test’ then will trigger a __coveragescanner_install() call just in the beginning, and __coveragescanner_clear() and __coveragescanner_testname() calls for each test case section which is being executed. Also it triggers __coveragescanner_teststate() and __coveragescanner_save() on each test case’s end.
  5. Additionally here is a small example which shows how to include this into your allready existing ’boost::test’ modules:
    To download: ObservedBoostTest.cpp pictures/zoom.png

  6. Compile the boost test with CoverageScanner activated.

42.6  xUnit

When Squish Coco is used with xUnit,5 it is possible to save the coverage for each single test. To do this, we define collections and hooks that are executed before and after each test.

The tests themself need not to be instrumented, but the code that is tested must be built with CoverageScanner. A small adaptation of the xUnit test suite must be made. These changes work even if the library that is tested is not instrumented, but in this case no code coverage information will be generated.

Proceed as follows:

  1. Add to your xUnit test library the following code:

    It provides a set of functions to handle the coverage information and to save the execution report (the .csexe file).

    To download: Coverage.cs pictures/zoom.png
  2. Add to your xUnit test library the following code:

    It contains a definition of a xUnit collection which saves the coverage after each unit test has been executed. If a collection is already present in the code, it is necessary to adapt the existing one with this file as guideline.

    To download: Hooks.cs pictures/zoom.png
  3. Activate the collection for each test class. To do it, just add “[Collection("SquishCoco")]” to the definition of each test class. For example:
    [Collection("SquishCoco")]
    public class Class2Tests
    {
        [Fact()]
        public void dummyTest()
        {
            Assert.True(false, "Dummy test");
        }
    }

After running the tests, a file coverage.csexe will be generated; it can then be imported into the corresponding coverage database (.csmes file). With CoverageBrowser, it is then possible to analyze the coverage of each test separately.

42.7  NUnit

Squish Coco provides an addin for NUnit6 version 2.4.4 and above as sample. To install it proceed as followings:

  1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj provided in the sample directory.
  2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.
  3. Start NUnit.exe and verify that the addin “NUnit Squish Coco” is loaded.

Once installed, as soon as NUnit’s test driver is executing a C# or C++ managed unit test test.dll, it generates automatically a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Squish Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application’s instrumentation database with CoverageBrowser or cmcsexeimport.

42.8  Catch2

Catch27 is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section.

Complete code example:

To download: Catch2Listener.cpp pictures/zoom.png

Follow these steps:

  1. Call __coveragescanner_install() and Catch::Session.run() in the main() function. Catch2 provides a predefined main(), which needs to be disregarded by defining only CATCH_CONFIG_RUNNER before including the Catch2 header:
        #define CATCH_CONFIG_RUNNER
        #include "catch.hpp"
  2. Create a Catch2 listener class - inheriting Catch::TestEventListenerBase - which records the code coverage of each section after it is completed. (Note: Testcases also count as sections in Catch2. There is no need to listen to testcase events specifically.)

    In the created listener class (CoverageScannerListener) we use the following member functions:

    sectionStarting()
    This function is called before each testcase and section begins.

    In it, we compute a test name with the information provided by Catch2 and pass it to the Squish Coco library with __coveragescanner_testname().

    We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored.

    sectionEnded()
    This function is called after a testcase or section has ended.

    It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  3. We add this listener by adding the following Catch2 macro:
      CATCH_REGISTER_LISTENER( CoverageScannerListener )
Remember to exclude any test framework sources from instrumentation (see Chapter 3.3)!

42.9  Squish

It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverage of a Squish test suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Squish Coco features the comparison of individual executions, including the calculation of an optimal order.

42.9.1  General Approach

The approach depicted below is based on the possibility to apply information about the name, the result and a free-form comment to each execution (stored in .csexe files). See Chapter 34.1 for the details.

As an example we’ll use Squish for Qt’s addressbook on a Unix-based system and a JavaScript test script.

  1. First we’ll initialize the execution data with the name of the Squish test case that is being run.
    function main()
    {
        var currentAUT = currentApplicationContext();
        var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
        var testCase = squishinfo.testCase;
        var testExecutionName = testCase.substr(testCase.lastIndexOf('/') + 1);
        var file = File.open(execution, "a");
        file.write("*" + testExecutionName + "\n");
        file.close();
        var ctx = startApplication("addressbook");
    ...
  2. Insert the main test script at this point
  3. After the main test script we’ll log the result of the test for the coverage tool:
    ...
        // wait until AUT shutdown
        while (ctx.isRunning) {
            snooze(1); // increase time if not enough to dump coverage data
        }

        // test result summary and status
        var positive = test.resultCount("passes");
        var negative = test.resultCount("fails") + test.resultCount("errors") + test.resultCount("fatals");
        var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
        var status = negative == 0 ? "PASSED" : "FAILED";
        var file = File.open(execution, "a");
        file.write("<html><body>" + msg + "</body></html>\n");
        file.write("!" + status + "\n")
        file.close();
    }

When you execute the scripts containing these steps, the Squish CocoExecution Report loads with the test case name, status and execution summary in the execution details and comments.

42.9.2  Simplified for Reuse

  1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:
    function getExecutionPath() {
        var currentAUT = currentApplicationContext();
        var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
        return execution;
    }

    function logTestNameToCocoReport(currentTestCase, execution) {
        var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf('\\') + 1);
        var file = File.open(execution, "a");
        file.write("*" + testExecutionName + "\n");
        file.close();
    }

    function logTestResultsToCocoReport(testInfo, execution){

        var currentAUT = currentApplicationContext();

        // wait until AUT shuts down
        while (currentAUT.isRunning)
          snooze(5);

        // collect test result summary and status
        var positive = testInfo.resultCount("passes");
        var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +
            testInfo.resultCount("fatals");
        var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
        var status = negative == 0 ? "PASSED" : "FAILED";

        // output results and status to Coco execution report file
        var file = File.open(execution, "a");
        file.write("<html><body>" + msg + "</body></html>\n");
        file.write("!" + status + "\n")
        file.close();
    }

    A Python version of this code is:

    import re

    def getExecutionPath():
        currentAUT = currentApplicationContext()
        execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name}
        return execution

    def logTestNameToCocoReport(currentTestCase, execution):
        testExecutionName = re.search(r'[^\\]\w*$', currentTestCase)
        testExecutionName = testExecutionName.group(0)
        file = open(execution, "a")
        file.write("*" + testExecutionName + "\n")
        file.close()

    def logTestResultsToCocoReport(testInfo, execution):
        currentAUT = currentApplicationContext()
        # wait until AUT shuts down
        while (currentAUT.isRunning):
            snooze(5)

        # collect test result summary and status
        positive = testInfo.resultCount("passes")
        negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals")
        msg = "TEST RESULTS - Passed: %(positive)s  |  Failed/Errored/Fatal: %(negative)s" % {'positive': positive, 'negative': negative}
        if negative == 0:
            status = "PASSED"
        else:
            status = "FAILED"

        # output results and status to Coco execution report file
        file = open(execution, "a")
        file.write("<html><body>" + msg + "</body></html>\n")
        file.write("!" + status + "\n")
        file.close()
  2. Add the following function calls after startApplication() in the main test script:
    execution = getExecutionPath();

    logTestNameToCocoReport(squishinfo.testCase);

    In Python:

    execution = getExecutionPath()

    logTestNameToCocoReport(squishinfo.testCase)
  3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the following function:
    logTestResultsToCocoReport(test);

    In Python:

    logTestResultsToCocoReport(test)
  4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.

Your main test script should be similar to the following:

source(findFile("scripts","squishCocoLogging.JS"))

function main()
{
    startApplication("addressbook");
    execution = getExecutionPath();
    logTestNameToCocoReport(squishinfo.testCase, execution);

    try {
        // body of script
    }
    catch(e) {
       test.fail('An unexpected error occurred', e.message)
    }
    finally {
       logTestResultsToCocoReport(test, execution)
    }
}

Python version:

source(findFile("scripts","squishCocoLogging.py"))

def main():
    startApplication("addressbook")
    execution = getExecutionPath()
    logTestNameToCocoReport(squishinfo.testCase, execution)

    try:
        try:
            # body of script
        except Exception, e:
            test.fail("test failed: ", e)
    finally:
        logTestResultsToCocoReport(test,execution)

Chapter 43  Installing Coco in Docker

In this chapter we give an example on how to run Squish Coco, terminal only interaction, in Docker.

43.1  Prerequisites

  1. A Linux machine, in this example Debian,
  2. Docker (latest version, but any stable version should work fine).

43.2  Installation of Docker

Before installing Squish Coco we need docker to be installed. To check if docker is running on your machine, simply type

$ docker --version

Skip the following step if docker is already installed on your machine.

43.2.1  Installation from the Debian repository

A standard Debian distribution contains already a Docker package. It is usually called docker or docker.io. To install it, run

$ sudo apt-get install docker.io

43.2.2  Installation from the Docker repository

If you need the newest version of Docker, you can use the Docker repository instead. The following procedure to install Docker is mostly the same for all Linux based machines. Please look into docker docs for more info, if necessary.

  1. Update the apt package index:
    $ sudo apt-get update
  2. Install packages to allow apt to use a repository over HTTPS:
    $ sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common
  3. Add Docker’s official GPG key:
    $ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
  4. Use the following command to set up the stable repository:
    $ sudo add-apt-respostiory \
            "deb [arch=amd64] https://download.docker.com/linux/debian \
            $(lsb_release -cs) \
            stable"
  5. Update the apt package index:
    $ sudo apt-get update
  6. Install the latest version of Docker CE and containerd:
    $ sudo apt-get install docker-ce docker-ce-cli containerd.io

43.2.3  Verification of the installation

Now verify that your user account has the right to run docker containers. On most UNIX® systems, a user must be in the docker group. To verify this, one can use the “id” command, which prints the current groups a user is in. If one is not member of docker, one can run

$ sudo adduser ⟨own accout⟩ docker

But afterwards it is usually necessary to log in again in order to make the group change visible.

To verify that Docker CE is installed correctly, run the hello-world image:

$ docker run hello-world

Note: If you want to remove the hello-world image, first run

$ docker ps -a

The output should be a list of containers, with name, status, and more. It should contain one header line and one line for the hello-world image. In the first column, there is the container ID of the hello-world image. Run

$ docker rm ⟨container_id

43.3  Installation of Coco in a Docker container

  1. Create an empty directory. We will refer to it as the work directory.
  2. Download a Coco package to the work directory. It must be a file that ends in “.run”, like “SquishCocoSetup_5.0.0_Linux_x86_64.run”.
  3. In the work directory, create a file named Dockerfile, with the following content:
    # Select base image
    FROM debian

    # Install packages for your project
    RUN apt-get update && apt-get upgrade -y && apt-get install -y libgtk2.0-dev

    # Set "secret" as root password
    RUN echo "root:secret" | chpasswd

    # Define a user
    RUN useradd -ms /bin/bash myapp
    USER myapp
    WORKDIR /home/myapp

    # Copy the Coco installer to the work directory
    COPY SquishCocoSetup*.run .
    For an actual project, the content of the file must be adapted. In this example, it does the following:
    1. Loading an image from the Docker Hub website. In this case it is the most recent Debian image, but any Linux image should work.
    2. Install some packages. In a real use case, these would be the prerequisites of the software project for which code coverage should be measured.

      The installation commands that are given here obviously only work on a Debian or Debian-like system, like Ubuntu.

    3. Setting a root password. This is later needed by the Squish Coco installer, which runs from a user account.
    4. Create a user account.
    5. Copy the Squish Coco installer into the container. It will there be located in the home directory of the user myapp.

      The wildcard pattern “SquishCocoSetup*.run” matches the name of any Squish Coco installer package, independent of the version number.

  4. In the work directory, run the command
    $ docker build -t squishcoco .
    Docker will then create an image with the name squishcoco.
  5. To run the image, execute the command
    $ docker run -it --name runner squishcoco
    Then a container named runner will be created and started.
  6. You will now see the command prompt of the container, no longer that of your operating system. You are now in the home directory of the user myapp.

    This directory also contains a copy of the Coco installer package. You can run it with a command like

    myapp@a6e8af:~$ bash SquishCocoSetup_5.0.0_Linux_x86_64.run

    The installer will then let you choose between different installation modes. Choose the first, “Installation on a the local machine”. The installer will then ask for the root password, which is of course “secret”.

    At the end of the installation, the installer asks, “Do you want to start Squish Coco License Manager in order to register this product?”. Answer with no, because the installer uses a graphical interface, which is usually not possible in a Docker container.

  7. Instead, the non-GUI version of the license manager mut be used. Run one of the following commands:
    • If you got an activation code, make sure that the Docker container has internet access enabled, then run
      myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --fetch-license-key=⟨activation code
      The license key will then be fetched from the license server. Afterwards, the internet access can be disabled again.
    • If you got a license key, run
      myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --license-key=⟨license key
    • If you use a license server, run
      myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --license-server=⟨server⟩:⟨port
      In this case, the Docker container needs to have network access enabled during compilation and whenever a Squish Coco tool runs. When the instrumented program runs, no network access is needed.
    If an error message is displayed, contact froglogic support.

    Alternatively, you can check if the license is activated or not by running the command below

    myapp@a6e8af:~$/opt/SquishCoco/bin/cocolic --check
  8. Now Squish Coco is installed in Docker. You can exit the container with the “exit” command.
  9. To enter the container again, run
    $ docker start -ai runner

Chapter 44  Atlassian Bamboo integration

44.1  Introduction

Atlassian Bamboo is a continuous integration server used to build, test and release software. The Squish Coco plug-in provides seamless integration with Bamboo. Users can add one or more Squish Coco tasks to a Bamboo job. Each Squish Coco task can perform the following actions:

  1. Import an execution report into an instrumentation data base.
  2. Merge several instrumentations databases.
  3. Generate a code coverage report.

The user can choose only one action to be performed, two actions or all three actions. If more than one action is chosen, they are executed in the order mentioned above.

pictures/manual.tmp071.png
Figure 44.1: Squish Coco task configuration overview

After the actions have ended, files that are created by Squish Coco (like the coverage report), can be stored to allow efficient troubleshooting.

44.2  Agent configuration

An agent is a service that runs Bamboo builds, tests and deployments. The “Agent Capabilities” setting lets users specify which agents are capable of running Squish Coco jobs. In order to execute it, we must first install Squish Coco at the agent host. Then we need to set up an agent-specific capability called ‘coco’ with Squish Coco installation directory as a value. (Go to the configuration entry “Bamboo administration|BUILD RESOURCES|Agents|AgentName|Agent-specific capabilities”.)

Later on, during the job configuration, we need to set the existence of a ‘coco’ capability as a job requirement. This way, Squish Coco jobs will be executed only on agents on which Squish Coco in installed.

44.3  Importing an execution report

This action calls cmcsexeimport to import the execution report (.csexe file) into an instrumentation data base (.csmes file). The user needs to provide an absolute or relative path (from the Bamboo build directory) to the .csexe and the .csmes file, and also a title for the execution.

When this action is activated, the following command is executed on the agent host:

cmcsexeimport --debug --title=⟨Title⟩ -m ⟨CSMesFile⟩ -e ⟨CSExeFile

44.4  Merging the instrumentation databases

This action calls cmmerge to merge several instrumentation databases (.csmes files). We need to provide the full or relative path to the resulting .csmes output file. If we provide a relative path (or just a filename), then the .csmes output file will be generated in the Bamboo working directory or a subdirectory relative to it. The .csmes input files need to be specified. Multiple files can be used, separated by a comma.

When this action is activated, the following command will be executed on the agent host:

cmmerge --verbose -o ⟨CSMesOutputFile⟩ ⟨CSMesInputFile1⟩ ... ⟨CSMesInputFileN

Additionally, the user can select the option ‘merge only instrumentations and executions present in the reference file’. This option is useful for importing .csmes files with unit tests. When the option is selected, the user must provide the name of a .csmes reference file. This will cause that above cmmerge command will be executed with additional option ‘-i’.

cmmerge --verbose -o ⟨CSMesOutputFile⟩ -i ⟨CSMesRefFile⟩ ⟨CSMesInputFile1⟩ ... ⟨CSMesInputFileN

44.5  Generating the coverage report

The last action calls cmreport to generate an HTML Report. We need to provide .csmes file name (either absolute or relative path). The .csmes file must contain the previously imported execution report (i.e. by using the first action offered by the plugin).

cmreport --title=⟨BambooJobName⟩ -m ⟨CSMesFile⟩ --debug --html=coco/report.html

The generated report will be stored in coco subdirectory, therefore we can define Artefact with ‘coco/**’ as the copy pattern. This allows us to view the report directly in Bamboo after job execution. The report title is built using a Bamboo job name.

44.6  Examples

44.6.1  A coverage report from a single execution

We need to activate first the action ‘Import an execution report into an instrumentation data base’ to import the execution report (.csexe file) into an instrumentation data base (.csmes). Finally, we need to activate the third action, ‘Generate code coverage report’. The .csexe file for this action is the same as for first action.

pictures/manual.tmp072.png
Figure 44.2: Squish Coco Task configuration for example addressbook application

44.6.2  Coverage report for unit tests

In this scenario we would like to generate coverage report for unit tests execution. To achieve this goal in Bamboo, we need a job that consists of two Squish Coco tasks.

After the build of the program which is tested, we have two .csmes files: one for our application and a second for the unit tests. We need to merge those instrumentation databases where the .csmes file for our application will be a reference during the merge. The merge step creates a new .csmes file as its output file. This is done in the following task:

pictures/manual.tmp073.png
Figure 44.3: Squish Coco Task configuration to merge unit test executions

In the second task, the execution report for the unit tests is imported into .csmes file (which was the .csmes output file of the previous task). The last action in the task, ‘Generate code coverage report’, generates an HTML report.

pictures/manual.tmp074.png
Figure 44.4: Squish Coco Task configuration for importing execution and generating report

Chapter 45  Jenkins CI integration

Jenkins CI is a popular continuous integration tool. Squish Coco can be run under Jenkins, and the changes of the coverage over time displayed by a plugin. This chapter describes the necessary setup.

45.1  Prerequisites

45.1.1  Getting a license

The Jenkins process runs under an own account, usually called “jenkins”. When therefore Squish Coco runs under Jenkins CI with a node-locked license, it needs a specific license for its account.

How to get this license is described in the following section; the section after that one describes the use of a license server.

Installing a node-locked license

In order to generate the license, the Squish Coco license tools need to be run from the “jenkins” account. There are two ways to do it, depending on whether the Jenkins server has an Internet connection.

Using a license server

For the use of Squish Coco with a license server, one has to create a new Jenkins CI project, which only runs (under Microsoft® Windows) the following command:

"Windows Coco\cocolic" --license-server=⟨host⟩:⟨port

where ⟨host⟩ and ⟨port⟩ refer to the license server (see Chapter 27). The colon and the port can be omitted if the default port is used.

Under Linux and macOS, this becomes

cocobin⟩/cocolic --license-server=⟨host⟩:⟨port

Then the license is installed.

45.1.2  Getting an EMMA plugin

Coco uses the EMMA-XML format to communicate the coverage results to Jenkins. Jenkins therefore needs a plugin to display EMMA data. We use here the plugin at https://wiki.jenkins-ci.org/display/JENKINS/Emma+Plugin. Install it in Jenkins with the Jenkins plugin manager.

45.2  Adaption of the project to Jenkins

We will assume here that you already compile and run your project under Jenkins and that you can also generate coverage information, but not yet under Jenkins.

With the license installed, code coverage should now work under Jenkins.

The build process must now be configured in such a way that a single .csmes file is generated that contains all the execution data. (This is done with cmcsexeimport.)

Now extend your build process such that at the end a report im EMMA-XML format is generated. This is done by calling

"Windows Coco\cmreport" -m project.csmes --emma=report.xml

where project.csmes must be replaced with the name of your results file.

We then use the plugin to display the data that were written to the file report.xml. To do this, select the configuration section of your Jenkins project and add a post-build action. Since the plugin is installed, the "Add post-build action" menu contains an entry "Record Emma coverage report". Select it, and in the field "Folders or files containing Emma XML reports" enter "report.xml".

Then after the next build, a coverage report will be generated.

pictures/manual.tmp075.png
Figure 45.1: Generating a coverage report from report.html with Jenkins CI.
email squish@froglogic.com