Menu

Squish Coco

Code Coverage Measurement for Tcl, QML, C# and C/C++

Part X
Coco Integration Handbook

Chapter 37  Support for IDEs and toolkits

In this chapter we give examples how Squish Coco can be combined with several IDEs and build environments.

37.1  GNU Makefiles

Mostly, in makefiles, the C and C++ compiler and the linker are defined using the environment variables CC, CXX and LINK. This can be substituted by CoverageScanner by setting CC, CXX and LINK in the command arguments of make.

Examplemake LINK=csg++ CXX=csg++ CC=csgcc

37.2  CygWin

To install the CoverageScanner compiler wrapper for GCC and G++ on CygWin, proceed as follows:

  1. Open the “Build Environment Selection” application. (Windows Coco\toolselector.exe)
  2. Select the item “CygWin - www.cygwin.com”.
  3. Click on “Install CygWin Support”. The build environment selection dialog then displays the list of generated compiler wrappers.

Open then the CygWin console and compile your application with csgcc instead of gcc (or csg++ instead of g++).

37.3  Scratchbox

If Squish Coco is installed on the root file system1, a compiler wrapper is created for each compiler supported by Scratchbox. To invoke CoverageScanner, prepend cs to the name of the cross-compiler.

37.4  Kitware CMake

CMake is a platform independent build tool from Kitware which can be downloaded from http://www.cmake.org.

When Squish Coco is used with CMake, the changes are partially dependent on the tool chain that is used for compilation. We will now first describe the addition of a new build type, which is independent from the tool chain, and then the additional changes for Microsoft® Visual Studio® and GNU GCC.

37.4.1  Adding new build type for instrumented compilation

The first step is independent of the toolchain that is used. Its purpose is to declare the CMake variables that are used to specify the instrumented compilation. In CMake this is done by declaring a build type, which we will here call COVERAGE.

To do this, add to the to CMakeLists.txt file the following lines. (The variable COVERAGE_FLAGS in the first line specifies the CoverageScanner command line options. Change its value to fit you needs. Only --cs-on must always be present.)

SET(COVERAGE_FLAGS "--cs-on --cs-count")
SET(CMAKE_CXX_FLAGS_COVERAGE
    "${CMAKE_CXX_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the C++ compiler during coverage builds."
    FORCE )
SET(CMAKE_C_FLAGS_COVERAGE
    "${CMAKE_C_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the C compiler during coverage builds."
    FORCE )
SET(CMAKE_EXE_LINKER_FLAGS_COVERAGE
    "${CMAKE_EXE_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used for linking binaries during coverage builds."
    FORCE )
SET(CMAKE_SHARED_LINKER_FLAGS_COVERAGE
    "${CMAKE_SHARED_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the shared libraries linker during coverage builds."
    FORCE )
SET( CMAKE_STATIC_LINKER_FLAGS_COVERAGE
    "${CMAKE_STATIC_LINKER_FLAGS_RELEASE} ${COVERAGE_FLAGS}" CACHE STRING
    "Flags used by the static libraries linker during coverage builds."
    FORCE )
MARK_AS_ADVANCED(
    CMAKE_CXX_FLAGS_COVERAGE
    CMAKE_C_FLAGS_COVERAGE
    CMAKE_EXE_LINKER_FLAGS_COVERAGE
    CMAKE_SHARED_LINKER_FLAGS_COVERAGE
    CMAKE_STATIC_LINKER_FLAGS_COVERAGE
    COMPILE_DEFINITIONS_COVERAGE
)

These commands take the compiler and linker flags of the “Release” build type and add to them the coverage flags. If you want to use instead the flags of another build type, replace the suffix “_RELEASE” in this code with the name of another build type, e.g. with “_DEBUG”.

37.4.2  Microsoft® Visual Studio®

Under Microsoft® Visual Studio®, we need to make the new Coverage build type visible to the IDE. To do this, add to CMakeLists.txt the lines

if(CMAKE_CONFIGURATION_TYPES)
   set(CMAKE_CONFIGURATION_TYPES Debug Release MinSizeRel RelWithDebInfo Coverage)
   set(CMAKE_CONFIGURATION_TYPES "${CMAKE_CONFIGURATION_TYPES}" CACHE STRING
     "Reset the configurations to what we need" FORCE)
endif()

If necessary, customize the list of configuration types according to your needs. To make the changes visible to Microsoft® Visual Studio®, a complex procedure is apparently needed:2

  1. Compile CMakeLists.txt with Ctrl-F7. If warnings occur, they can be ignored.
  2. Make a trivial change in CMakeLists.txt, like adding a space.
  3. Compile again. Then a dialog, "File modification detected", should appear. Click "Reload".

Then the list of solution configurations is updated and one can compile in coverage mode.

37.4.3  Compilation with Microsoft® NMake

In a project that is compiled whith NMake, the following must be done.

  1. Create a toolchain definition file cl.cmake which replaces the compiler and linker with their CoverageScanner wrappers.

    Example

    # this one is important
    SET(CMAKE_SYSTEM_NAME Windows)

    # specify the cross compiler
    FILE(TO_CMAKE_PATH "$ENV{SQUISHCOCO}/visualstudio" SQUISHCOCO)
    SET(CMAKE_C_COMPILER ${SQUISHCOCO}/cl.exe
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_CXX_COMPILER ${SQUISHCOCO}/cl.exe
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_LINKER ${SQUISHCOCO}/link.exe
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
  2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE.

    Example

    cmake.exe -DCMAKE_TOOLCHAIN_FILE=cl.cmake -DCMAKE_BUILD_TYPE=COVERAGE \
              -G "NMake Makefiles" ⟨path of cmake project
  3. Build the project with nmake.

37.4.4  Compilation with GNU GCC

The following must be done in a project that is comiled with gcc.

  1. Create a toolchain definition file gcc.cmake which replaces the compiler and linker with their CoverageScanner wrappers.

    Example

    find_program(CODE_COVERAGE_GCC csgcc)
    find_program(CODE_COVERAGE_GXX csg++)
    find_program(CODE_COVERAGE_AR csar)

    # specify the cross compiler
    SET(CMAKE_C_COMPILER "${CODE_COVERAGE_GCC}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_CXX_COMPILER "${CODE_COVERAGE_GXX}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_LINKER "${CODE_COVERAGE_GXX}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
    SET(CMAKE_AR "${CODE_COVERAGE_AR}"
        CACHE FILEPATH "CoverageScanner wrapper" FORCE)
  2. Create a Makefile project. Set the toolchain to the CoverageScanner wrapper and the build mode to COVERAGE.

    Example

    cmake -DCMAKE_TOOLCHAIN_FILE=gcc.cmake -DCMAKE_BUILD_TYPE=COVERAGE \
          -G "Unix Makefiles" ⟨path of cmake project
  3. Build the project with make.

37.5  Qt Framework

37.5.1  Qt Creator

The following steps are needed to add instrumentation to a Qt Creator project.

With these settings, a new build mode CodeCoverage has been created that creates an instrumented version of your program.

If the project has unit tests, it is possible to execute them at every build and also generate a coverage report for them. In order to do this, the following additional steps are needed.

Now, every time you click on Build (not just Run) with this configuration, the project will be built, instrumented and optionally a report will be generated. Note that in case the project has been built before this configuration was performed, it will be required to execute the Clean Build step before building, so that the coverage measurement data will be generated.

Also keep in mind that when there was an error and one of the build steps exits with a non-zero exit status code, Qt Creator stops the build and displays the error message of the executable that failed, instead of continuing the build. This means that if a unit test fails, instead of cmcsexeimport running and importing the execution as a failed execution, cmcsexeimport does not run at all until the test case failure was fixed.

37.5.2  qmake

qmake is a tool to generate a Makefile in a platform-independent way, using as specification a so-called “project file”. By default, qmake chooses the programs that are used for compilation. We can however use the Squish Coco wrappers by setting some of qmake’s variables to new values. This can be done via command line parameters or by editing the project configuration files.

Command line.

In simple cases, we can do this on the command line. There are two methods.

Project configuration.

In more complex cases it is better to enable coverage by putting some declarations in the qmake project files. Since one needs to build the project with and without coverage, the new definitions must be put into a scope. (A scope is a region in a qmake project file that can be activated on demand.)

The following listing shows a template for such a scope, named CoverageScanner, which should be sufficient for most most projects.

CodeCoverage {
  COVERAGE_OPTIONS =

  QMAKE_CFLAGS   += $$COVERAGE_OPTIONS
  QMAKE_CXXFLAGS += $$COVERAGE_OPTIONS
  QMAKE_LFLAGS   += $$COVERAGE_OPTIONS

  QMAKE_CC=cs$$QMAKE_CC
  QMAKE_CXX=cs$$QMAKE_CXX
  QMAKE_LINK=cs$$QMAKE_LINK
  QMAKE_LINK_SHLIB=cs$$QMAKE_LINK_SHLIB
  QMAKE_AR=cs$$QMAKE_AR
  QMAKE_LIB=cs$$QMAKE_LIB
}

Here we also set the variables QMAKE_LINK_SHLIB and QMAKE_AR, which contain the names of the command to link shared libraries and that to generate archives. Furthermore, can use COVERAGE_OPTIONS to set coveragescanner commandline options (see Chapter 9.2) to customize the project. An empty value for COVERAGE_OPTIONS will also work and results in a default instrumentation.

In a small project, the CodeCoverage scope may then be copied to all profile files of the project, i.e. those that end in “.pro”. (In fact, it is enough to insert them only into those files that actually compile code and not those that only include other files.) If the project is larger, it has very often a file with common settings that is included by all profiles: This is then the most convenient place to insert the CodeCoverage scope only once.

The new coverage scope is by default inactive. To enable code coverage in a project you want to built, just add the name of the scope to the CONFIG variable when configuring it with qmake:

$ qmake CONFIG+=CodeCoverage

37.5.3  moc

The Meta-Object Compiler moc adds automatically new methods to each classes derived from QObject. For example, the translation function tr, the source code for all signals, the cast operator qt_cast, …In order to instrument the code using the Qt Framework and not the Qt code itself, CoverageScanner provides the command line options --cs-qt3 for Qt3 and --cs-qt4 for Qt4 and Qt5. They are enabled by default.

In this case:

37.5.4  qbs

To use CoverageScanner with the Qt Build Suite (qbs) it can be set up as a toolchain. This toolchain can then be used for all qbs projects.

In order to set up CoverageScanner as a toolchain, issue the following command:

qbs setup-toolchains --type gcc /opt/SquishCoco/bin/csgcc csgcc

For Unix-based operating systems some additional configuration steps are necessary:

qbs config profiles.csgcc.cpp.archiverPath /opt/SquishCoco/bin/csar
qbs config profiles.csgcc.cpp.linkerName csg++
qbs config profiles.csgcc.cpp.nmPath /opt/SquishCoco/bin/csgcc-nm

The csgcc toolchain can then also be used as base profile for Qt projects:

qbs setup-qt /opt/Qt/bin/qmake qt-csgcc
qbs config profiles.qt-csgcc.baseProfile csgcc

37.6  SCons

To use Squish Coco with SCons, proceed as followings:

Here is a code snippet which can be used for Microsoft® Visual Studio® command line tools:

import os
from os.path import pathsep

env = Environment()

# Add the path of Squish Coco compiler wrapper
env[ 'ENV' ][ 'PATH' ] = os.environ[ 'SQUISHCOCO' ] + pathsep + env[ 'ENV' ][ 'PATH' ]
# TEMP variable need to be defined
env[ 'ENV' ][ 'TEMP' ] = os.environ[ 'TEMP' ]

# Set the compiler to Squish Coco wrappers
env[ 'CC' ]   = 'cs' + env[ 'CC' ] ;
env[ 'AR' ]   = 'cs' + env[ 'AR' ] ;
env[ 'LINK' ] = 'cs' + env[ 'LINK' ] ;

# Code coverage settings
coverageflags = [ '--cs-count' ]
env[ 'CCFLAGS' ]   = env[ 'CCFLAGS' ] + coverageflags ;
env[ 'ARFLAGS' ]   = env[ 'ARFLAGS' ] + coverageflags ;
env[ 'LINKFLAGS' ] = env[ 'LINKFLAGS' ] + coverageflags ;

37.7  ARM® Keil® µVision

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper for ARM and enable the code coverage analysis during the compilation.

Processed as followings:

  1. Select the Squish Coco tool chain instead of the original ARM toolchain.
    1. On the ARM® Keil® µVision IDE click on "Project->Manage->Components, Environments, Books…".
    2. Click on "Folders/Extensions", and replace .\ARMCC\bin through .\SquishCoco\bin in the field "ARMCC Folder:".
      pictures/manual.tmp030.png
      Figure 37.1: Installation of CoverageScanner on ARM® Keil® µVision: Setting the path of CoverageScanner.
  2. Activate the code coverage analysis during the compilation
    1. Click on "Project->Option for Target…"
    2. On the "C/C++" tab, add --cs-on into the "Misc Controls" field.
      pictures/manual.tmp031.png
      Figure 37.2: Activating the code coverage analysis for an ARM® compiler
    3. On the "Linker" tab, add --cs-on into the "Misc Controls" field.
      pictures/manual.tmp032.png
      Figure 37.3: Activating the code coverage analysis for an ARM® linker

37.8  Green Hills® Software MULTI Launcher

37.8.1  Installation

To install CoverageScanner compiler wrapper for the Green Hills® Software tool chain, proceed as follows:

  1. Open the “Build Environment Selection” application. (Windows Coco\toolselector.exe)
  2. Select the item “Green Hills”.
  3. Select the directory o which the Green Hills® Software compilers are installed
  4. Click on “Install Green Hills Support” and wait for the confirmation dialog that the tools are generated.

37.8.2  Command Line Tools

The command line tools for the Green Hills® Software compiler are installed in the folder squishcoco of the installed native tool chain. Example: if the native tool chain is installed under c:\ghs\comp_201426, the Squish Coco compiler wrapper are installed under c:\ghs\comp_201426\squishcoco.

The compiler wrappers are replacing completely the Green Hills® Software tool chain. To activate the code coverage analysis, it is necessary to add the parameter --cs-on to the compiler arguments.

37.8.3  CoverageScanner Library

In order to save an execution report, it is necessary to:

  1. provide to the CoverageScanner library a list of I/O function which permits to upload the coverage information to the host.
  2. Save manually the coverage report on a specific trigger.

To save a coverage report the following code snippet can be used (an this code in an event handler which should trigger the report generation):

   1 #ifdef __COVERAGESCANNER__
   2     __coveragescanner_save();
   3     __coveragescanner_clear();
   4 #endif

To provide the i/O functions it is necessary to call __coveragescanner_set_custom_io() in the main() function of the application. At least three functions need to be provided by __coveragescanner_set_custom_io():

  1. An equivalent function of fopen() which opens the execution report file (with the extension .csexe) in append mode.
  2. An equivalent function of fclose()
  3. An equivalent function of fputs() to transfer the contain.

Example:  The following code write an execution report to the local file system using the C file API.

   1 #ifdef __COVERAGESCANNER__
   2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }
   3 static void *csfopenappend(const char *path)    { return (void*)fopen(path,"a+");  }
   4 static int csfclose(void *fp)                   { return fclose((FILE*)fp);        }
   5 #endif
   6 
   7 int main()
   8 {
   9 #ifdef __COVERAGESCANNER__
  10     __coveragescanner_set_custom_io( NULL,
  11             csfputs,
  12             csfopenappend,
  13             NULL,
  14             NULL,
  15             csfclose,
  16             NULL);
  17 #endif
  18 ....
  19 }

37.9  VisualDSP®

To enable the code coverage analysis it is first necessary to compile the project with Squish Coco compiler wrapper and enable the code coverage analysis during the compilation in the VisualDSP® configuration.

Processed as followings:

  1. Click on "Project->Project Options…"
  2. On the "Compile" tab, add --cs-on into the "Additional options" field.
    pictures/manual.tmp033.png
    Figure 37.4: Activating the code coverage analysis for the compiler
  3. On the "Link" tab, add --cs-on into the "Additional options" field.
    pictures/manual.tmp034.png
    Figure 37.5: Activating the code coverage analysis for the linker
  4. Rebuild your project.

Like for all embedded targets, it is necessary to provide a dedicated I/O since that in most of the cases no file systems are available for storing the code coverage information. Also since that embedded applications are generally not exiting, it is also necessary to implement a event handler which saves the execution report upon the reception of a specific trigger.

The simulator emulates the support of a file system. To save the coverage report on the current build directory, it is only necessary to register in the first lines of the main() a custom file I/O which uses the standard C file API.

Example

   1 #ifdef __COVERAGESCANNER__
   2 static int csfputs(const char *s, void *stream) { return fputs(s, (FILE *)stream); }
   3 static void *csfopenappend(const char *path)    { return (void*)fopen(path,"a+");  }
   4 static int csfclose(void *fp)                   { return fclose((FILE*)fp);        }
   5 #endif
   6 
   7 int main()
   8 {
   9 #ifdef __COVERAGESCANNER__
  10     __coveragescanner_set_custom_io( NULL,
  11             csfputs,
  12             csfopenappend,
  13             NULL,
  14             NULL,
  15             csfclose,
  16             NULL);
  17 #endif
  18 ....
  19 }

To record a coverage report when a specific trigger occurs, add the following source code lines in its handler:

   1 #ifdef __COVERAGESCANNER__
   2     __coveragescanner_save();
   3     __coveragescanner_clear();
   4 #endif

37.10  Microsoft® Visual Studio®

Squish Coco provides a wrapper for link.exe and cl.exe located on the %SQUISHCOCO%\visualstudio directory. It behaves exactly like the corresponding Microsoft® wrapper except that the code coverage analysis becomes activated when the option --cs-on is added to the command arguments. These wrappers call the Microsoft® tools for compilation or for linkage.

37.10.1  Microsoft® Visual Studio® .NET C# Compiler

To activate the instrumentation of C# source code, it is only necessary to add the define COVERAGESCANNER_COVERAGE_ON in the properties of the Microsoft® Visual Studio® .NET project. Other defines can be appended to select additional instrumentation options. The full list can be found on the chapter 9.3.

37.10.2  Microsoft® Visual Studio® .NET C and C++ Compiler

To use Squish Coco with Microsoft® Visual Studio® .NET proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.
    For Microsoft® Visual Studio® 2005 or 2008:
    1. Start Microsoft® Visual Studio® 2005 or 2008.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Projects->VC++ Directories".
    4. Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 37.6)
      pictures/manual.tmp035.png
      Figure 37.6: Installation of CoverageScanner on Microsoft® Visual Studio® 2005 and 2008: Setting the path of CoverageScanner.

    For Microsoft® Visual Studio® 2010:

    1. Start Microsoft® Visual Studio® 2010.
    2. Open a C++ project.
    3. Open the project properties using the context menu of the laded project.
    4. Select the item "Configuration Properties->VC++ Directories".
    5. Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of executable directories. (see Figure 37.7)
      pictures/manual.tmp036.png
      Figure 37.7: Installation of CoverageScanner on Microsoft® Visual Studio® 2010: Setting the path of CoverageScanner.
  2. To activate code coverage analysis:
    1. Open a Visual C or C++ project.
    2. Edit the project settings (click on "Project->Properties").
    3. Add to the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 37.8)
    4. In the additional arguments of the linker, add the --cs-libgen which permits to specify which library should be used for the generation of the CoverageScanner library. The table 37.1 contains the list of recommended settings.
    5. For Microsoft® Windows CE applications, append to the linker arguments the command line option --cs-architecture which permits to specify the target platform. The table 37.2 contains the list of available architectures.
LibraryLibrary FileCommand line option
Single ThreadedLIBC.LIB--cs-libgen=/ML
Static MultiThreadLIBCMT.LIB--cs-libgen=/MT
Dynamic Link (DLL)LIBCRT.LIB--cs-libgen=/MD
Debug Single ThreadedLIBCD.LIB--cs-libgen=/MLd
Debug Static MultiThreadLIBCMTD.LIB--cs-libgen=/MTd
Debug Dynamic Link (DLL)LIBCRTD.LIB--cs-libgen=/MDd
Table 37.1: CoverageScanner library settings for Microsoft® Visual Studio®
Targeted ArchitectureCommand line option
ARM Microprocessor--cs-architecture=ARM
ARM Microprocessor (Thumb code)--cs-architecture=THUMB
x86 Microprocessor--cs-architecture=IX86
MIPS16 Microprocessor--cs-architecture=MIPS16
MIPS Microprocessor--cs-architecture=MIPS
MIPS Microprocessor with FPU--cs-architecture=MIPSFPU
SH3 Microprocessor with FPU--cs-architecture=SH3
SH4 Microprocessor with FPU--cs-architecture=SH4
Table 37.2: List of target architectures
pictures/manual.tmp037.png
Figure 37.8: Activation of the instrumentation under Visual Studio® .NET.
pictures/manual.tmp038.png
Figure 37.9: Activation of the instrumentation under Visual Studio® .NET.

37.10.3  Squish Coco Add-In for Microsoft® Visual Studio®

Squish Coco provides for Microsoft® Visual Studio® 2005 up to Microsoft® Visual Studio® 2013 an Add-In which does automatically the steps described in the section 37.10.2. The Add-In itself is normally not mandatory to build an instrumented application.

Two different workflow are supported:

To use it:

  1. Open a C++ or a C# project.
  2. Click on “Tools->Code Coverage Build Mode…”.
  3. Select the base configuration and the project which should be compiled with code coverage support.
  4. And finally:
    • Select “Modify” and click on “Enable code coverage” to alter the current build mode and add the code coverage support. “Disable code coverage” permits removes all CoverageScanner command line options.
    • Select “New” and click on “Create new configuration” to generate a new build mode with code coverage support.
pictures/manual.tmp039.png
Figure 37.10: Microsoft® Visual Studio® Add-In
For Microsoft® Visual Studio® 2010, it might be necessary to add manually the path Squish Coco executables to the search path. See section 37.10.2 for more the detailed procedure.

37.10.4  Microsoft® Visual C++ Express

To use Squish Coco with Microsoft® Visual C++ Express proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the VC++ Directories.
    1. Start Microsoft® Visual C++ Express.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Projects->VC++ Directories".
    4. Add the entry $(SQUISHCOCO)\visualstudio to the first position in the list of directories. (see Figure 37.11)
      pictures/manual.tmp040.png
      Figure 37.11: Installation of CoverageScanner on Visual C++ Express: Setting the path of CoverageScanner.
  2. The activation of the code coverage analysis is similar to Microsoft® Visual Studio® .NET. (see Chapter 37.10.2)

37.10.5  Microsoft® Visual Studio® 6.0

To use Squish Coco with Microsoft® Visual Studio® 6.0 proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.
    1. Start Microsoft® Visual Studio® 6.0.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Directories".
    4. Select "Executable files" in the combobox "Show directories for:".
    5. Add the path of the directory visualstudio of the Squish Coco installation3 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\visualstudio, (see Figure 37.12))
      pictures/manual.tmp041.png
      Figure 37.12: Installation of CoverageScanner on Visual Studio® 6.0: Setting the path of CoverageScanner.
  2. To activate the code coverage analysis:
    1. Open a Visual C or C++ project.
    2. Edit the project settings (click on "Project->Properties").
    3. Add the option --cs-on to the additional command line arguments of the C or C++ compiler and linker. (see Figure 37.13)
    4. In the additional arguments of the linker, add the --cs-libgen which permits to specify which library should be used for the generation of the CoverageScanner library. The table 37.1 contains the list of recommended settings.
pictures/manual.tmp042.png
Figure 37.13: Activation of the instrumentation under Visual Studio® 6.0.
pictures/manual.tmp043.png
Figure 37.14: Activation of the instrumentation under Visual Studio® 6.0.

37.11  Microsoft® eMbedded Visual C++®

To use Squish Coco with Microsoft® eMbedded Visual C++®proceed as follows:

  1. Add the location of the CoverageScanner wrappers to the first position in the executable directories.
    1. Start Microsoft® eMbedded Visual C++®.
    2. Open the option dialog: click on "Tools->Preferences…".
    3. Select the item "Directories".
    4. Select "Executable files" in the combobox "Show directories for:".
    5. Select "Platform" and the targeted "CPUs".
    6. Add the path of the directory WinCE of the Squish Coco installation4 to the first position in the list of directories. (example: if Squish Coco is installed on c:\programme\SquishCoco, add the path c:\programme\SquishCoco\WinCE, (see Figure 37.15))
      pictures/manual.tmp044.png
      Figure 37.15: Installation of CoverageScanner on eMbedded Visual C++®: Setting the path of CoverageScanner.
  2. To activate the code coverage analysis:
    1. Open a Visual C or C++ project.
    2. Edit the project settings (click on "Project->Properties").
    3. Add the option --cs-on to the additional command line arguments of the C and C++ compiler and linker. (see Figure 37.16)
pictures/manual.tmp045.png
Figure 37.16: Activation of the instrumentation under eMbedded Visual C++®.
pictures/manual.tmp046.png
Figure 37.17: Activation of the instrumentation under eMbedded Visual C++®.

37.12  Eclipse IDE for C/C++

In Eclipse IDE for C/C++, code coverage is enabled for a configuration by replacing the names of the compilers it uses with the names of the Squish Coco compiler wrappers. This can be done in the following way:

  1. Start Eclipse.
  2. Load the C or C++ project that should be instrumented.
  3. Open the property window (Project-> Properties).
  4. Click on "C/C++ Build/Settings".
  5. Create a new configuration by clicking on "Manage Configurations...", and select it.
  6. Click on "Tools Settings" tab.
  7. Click on "GCC C++ Compiler" and prepend cs to the name of the compiler.
  8. Click on "GCC C Compiler" and prepend cs to the name of the compiler.
  9. Click on "C++ Linker" and prepend cs to the name of the linker.
  10. If it is a library project, also click on "GCC archiver" and replace the name of the archiver command ar with csar.

Now the Eclipse will use the Squish Coco wrappers instead of the compilers when compiling.

pictures/manual.tmp047.png
Figure 37.18: Eclipse settings

However, the Squish Coco wrappers are not by default in the search part and can not yet be found during compilation.

To change this, a copy of the PATH variable needs to be added to the "C/C++ Build->Environment" section of the property window, and the path of the Squish Coco binaries added. Under UNIX®, and with Squish Coco installed at the default location, PATH will then have a value like “/opt/SquishCoco/bin:/usr/local/bin:/usr/bin:/bin”.

Almost always it will be necessary to modify the behavior of Squish Coco by setting command line options. The easiest way to do this is by adding the variable COVERAGESCANNER_ARGS in "C/C++ Build->Environment". Its value then consists of command line options (see Chapter 9.2). Code instrumentation is however already activated if this variable is not set or empty.

If the value of COVERAGESCANNER_ARGS has changed, it is necessary to compile the whole project again; otherwise the new options have no effect.

If everything is done correctly and the project is compiled, one will see in the console window that csg++ is used instead of g++, etc., and that a .csmes file is created next to the place of the newly-built binary. When the binary then is run, a .csexe file is created in its working directory, which is typically a different directory from that of the .csmes file.

37.13  Apple® Xcode

To use Squish Coco with Apple® Xcode proceed as follows: To activate the code coverage analysis:

  1. Open a terminal window and set the CPLUSPLUS, LDPLUSPLUS, LD and CC to CoverageScanner compiler wrapper. The path of native compiler (clang, clang++, gcc or g++) need to be present in the PATH environment variable. Start Xcode using the open command.

    If GCC is used as compiler:

    SQUISHCOCO=/Applications/SquishCoco/wrapper
    export CC=$SQUISHCOCO/gcc
    export LD=$SQUISHCOCO/gcc
    export CPLUSPLUS=$SQUISHCOCO/g++
    export LDPLUSPLUS=$SQUISHCOCO/g++

    open /Developer/Applications/Xcode.app

    If clang is used as compiler:

    SQUISHCOCO=/Applications/SquishCoco/wrapper
    export CC=$SQUISHCOCO/clang
    export LD=$SQUISHCOCO/clang
    export CPLUSPLUS=$SQUISHCOCO/clang++
    export LDPLUSPLUS=$SQUISHCOCO/clang++
    export XCODE_TOOLCHAIN_DIR=/Applications/Xcode.app/Contents/Developer/Toolchains
    export XCODE_TOOLCHAIN=$XCODE_TOOLCHAIN_DIR/XcodeDefault.xctoolchain/usr/bin/

    export PATH=$XCODE_TOOLCHAIN:$PATH

    open /Applications/Xcode.app
  2. Open a Xcode C or C++ project.
  3. Edit the project settings (click on "Project->Edit Project Settings").
  4. Add the option --cs-on to the additional command line arguments of the C and C++ compiler (fields Other C Flags and Other C++ Flags) and linker (field Other Linker Flags). (see Figure 37.19)
  5. Disable the usage of the precomiled header: Open the settings of the active target ("Project->Edit Active Target") and remove the contains of Prefix Header.
pictures/manual.tmp048.png
Figure 37.19: Activation of the instrumentation under Apple® Xcode.

37.14  Wind River support

37.14.1  VxWorks support on Linux

To activate Squish Coco on Wind River’s Workstation proceed as follows:

  1. Start Wind River’s Workbench
  2. Select your project in the "Project Explorer" window and click on the entry "Properties" in the context menu.
  3. Select the build properties, the compiler tool chain and click on the "Variables" tab.
  4. Search for the TOOL_PATH variable and replace the /bin string at the end with /squishcoco:
    pictures/manual.tmp049.png
    Figure 37.20: Setting the TOOL_PATH variable

After this change, the project can be compiled with CoverageScanner or the native tool chain. To use CoverageScanner, it is necessary to add the option --cs-on to the command line arguments of the compiler and the linker.

To do this, proceed as followings:

  1. Start the Wind River Workbench
  2. Select your project in the "Project Explorer" window and click on "Properties" of the context menu.
  3. Select the build properties, the compiler tool chain and click on the "Tools" tab.
  4. Select the entry "C-Compiler" in the "Build Tool" combo box and add to the content of the "Tool Flags..." field the argument --cs-on:
    pictures/manual.tmp050.png
    Figure 37.21: Activating the code coverage analysis
  5. Do the same for the “C++-Compiler” and the “Linker” tool.

With these settings, a file with the extension .vxe.csmes is generated when the code is compiled. It contains the complete instrumented code and can be inspected with CoverageBrowser. The resulting target application will then create a file with the suffix .csexe after it has run.

This file is created on the target file system. It can them be transferred to the host and imported into the .vxe.csmes file to create a report.



A video that illustrates the content of this chapter is available on https://youtube.com/watch?v=#1.

Code Coverage with Squish Coco of a WxWorks application on Linux

Chapter 38  Support for specific test frameworks

38.1  CppUnit

CppUnit1 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

The following code is an example how this can be done:

To download: CppUnitListener.cpp pictures/zoom.png

In the example, we have done the following steps:

  1. We write a CppUnit listener class which records the code coverage of unit each test after it is completed.

    We want to be able to run the program with and without Squish Coco. Therefore we use in the code the macro __COVERAGESCANNER__ for conditional compilation. The macro is defined in every file that is instrumented by Squish Coco, without the need to #include anything.

    In the listener class, CppUnitListener, we use the following menber functions:

    startTest()
    This function is called before each test begins.

    In it, we compute a test name with the information provided by CppUnit and pass it to the Squish Coco library with __coveragescanner_testname().

    We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored.

    addFailure()
    This function is called after a test fails. It just sets a flag that is used by the other functions.
    endTest()
    This function is called after a test has ended.

    It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  2. We call __coveragescanner_install() in the main() function.
  3. We add this listener in the test manager of CppUnit, the class CPPUNIT_NS::TestResult. In the example above, this is done by the following lines:
    CoverageScannerListener coveragescannerlistener;
    controller.addListener( &coveragescannerlistener );

38.2  QTestLib

QTestLib is a unit test framework for Qt. It can easily be adapted to get the code coverage for each unit test.

Proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a subclass of QObject, named TestCoverageObject. It must record the code coverage at the end of every unit test.
  3. Instead of inheriting from QObject, let all your test cases inherit from TestCoverageObject.
  4. The TestCoverageObject class provides its own init() and cleanup() slots, which use the CoverageScanner API to save the code coverage report. If these slots are also declared in the test case classes, it is necessary to rename them to initTest() and cleanupTest().
  5. Compile your project with code code coverage enabled.

TestCoverageObject header:

To download: testcoverageobject.h pictures/zoom.png

TestCoverageObject source:

To download: testcoverageobject.cpp pictures/zoom.png

38.3  GoogleTest

GoogleTest2 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

Simply proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Write a TestEventListener class which records the code coverage report upon every unit test completion. The listener should set the name (using __coveragescanner_testname()) and clear the instrumentation (using __coveragescanner_clear()) before executing a test item (class member startTest()) to ensure to get only the coverage data of the concerned test. When an test item is executed, the instrumentation and the execution status should be saved (using __coveragescanner_teststate() and __coveragescanner_save()) in the class member endTest(). The class CodeCoverageListener give an implementation example.
  3. Add this listener in the Append function of the GoogleTest listener (function ::testing::UnitTest::GetInstance()->listeners().Append()).
  4. Compile the unit test using CoverageScanner.
To download: googletest.cpp pictures/zoom.png

38.4  CxxTest

CxxTest3 is a unit test framework for C++. This environment can easily be adapted to get the code coverage from each unit test.

Proceed as follows:

  1. Call __coveragescanner_install() in the main() function.
  2. Create a CxxTest TestListener class CoverageScannerListener by subclassing an existing listener.

    In the example below this is ErrorPrinter. It will record the code coverage report upon every unit test completion. To ensure to get only the coverage data of the concerned test, the listener should set the name with __coveragescanner_testname() and clear the instrumentation with __coveragescanner_clear() before executing a test item (class member enterTest()).

    When an test item is executed, the instrumentation and the execution status should be saved in the member function leaveTest() with __coveragescanner_teststate() and __coveragescanner_save().

    Finally, all test failure members of this class must be reimplemented to record the test failures.

  3. In the main() function call the run() function of CoverageScannerListener instead of CxxTest::ErrorPrinter().run().
  4. Compile the unit test with CoverageScanner activated.

Example

To download: CxxTestListener.cpp pictures/zoom.png

38.5  boost::test

boost::test4 is a unit test framework for C++ which is part of the boost libraries.

Proceed as follows:

  1. Implement a TestObserver, which derives from 'boost::unit_test_framework::test_observer' and store as "BoostTestObserver.hpp".
  2. Also implement (within the same file) a ’boost::test::fixture’ and define it to be executed in the beginning and in the end of each executed boost::test by: BOOST_GLOBAL_FIXTURE(FixtureName). The file then should look something similar to this:
    To download: BoostTestObserver.hpp pictures/zoom.png

  3. If it is included by any of your allready existing ’boost::test’ unit test .cpp files (must be done right after the in ’boost::test’ mandatory  BOOST_TEST_MODULE definition), It is being instantciated and initialized by the boost::test automatically.
  4. The ’boost::test’ then will trigger a __coveragescanner_install() call just in the beginning, and __coveragescanner_clear() and __coveragescanner_testname() calls for each test case section which is being executed. Also it triggers __coveragescanner_teststate() and __coveragescanner_save() on each test case’s end.
  5. Additionally here is a small example which shows how to include this into your allready existing ’boost::test’ modules:
    To download: ObservedBoostTest.cpp pictures/zoom.png

  6. Compile the boost test with CoverageScanner activated.

38.6  NUnit

Squish Coco provides an addin for NUnit5 version 2.4.4 and above as sample. To install it proceed as followings:

  1. Build NUnitSquishCoco.dll using the Microsoft® Visual Studio® project NUnitSquishCoco.vsproj provided in the sample directory.
  2. Copy NUnitSquishCoco.dll to the addins folder located in the bin folder where NUnit.exe can be found.
  3. Start NUnit.exe and verify that the addin “NUnit Squish Coco” is loaded.

Once installed, as soon as NUnit’s test driver is executing a C# or C++ managed unit test test.dll, it generates automatically a code coverage execution report test.dll.csexe automatically if test.dll is instrumented with Squish Coco. The code coverage information is organized into a tree containing the coverage and the execution status for each single unit test. The execution report can be then imported into the application’s instrumentation database with CoverageBrowser or cmcsexeimport.

38.7  Catch2

Catch26 is a unit test framework for C++ which can be easily adapted to get the code coverage from each unit test section.

Complete code example:

To download: Catch2Listener.cpp pictures/zoom.png

Follow these steps:

  1. Call __coveragescanner_install() and Catch::Session.run() in the main() function. Catch2 provides a predefined main(), which needs to be disregarded by defining only CATCH_CONFIG_RUNNER before including the Catch2 header:
        #define CATCH_CONFIG_RUNNER
        #include "catch.hpp"
  2. Create a Catch2 listener class - inheriting Catch::TestEventListenerBase - which records the code coverage of each section after it is completed. (Note: Testcases also count as sections in Catch2. There is no need to listen to testcase events specifically.)

    In the created listener class (CoverageScannerListener) we use the following member functions:

    sectionStarting()
    This function is called before each testcase and section begins.

    In it, we compute a test name with the information provided by Catch2 and pass it to the Squish Coco library with __coveragescanner_testname().

    We also call the function __coveragescanner_clear(): It empties the internal database and so makes sure that the coverage of the code that was executed before this test is ignored.

    sectionEnded()
    This function is called after a testcase or section has ended.

    It uses __coveragescanner_teststate() to record the execution status ("PASSED" or "FAILED") and then saves the code coverage report itself with __coveragescanner_save().

  3. We add this listener by adding the following Catch2 macro:
      CATCH_REGISTER_LISTENER( CoverageScannerListener )
Remember to exclude any test framework sources from instrumentation (see Chapter 3.3)!

38.8  Squish

It is easily possible to run the GUI testing tool Squish together with Squish Coco in order to get the C/C++coverage of a Squishtest suite. A more in-depth analysis is then possible, correlating each test case (and its results) with the respective coverage information. This is especially true since Squish Coco features the comparison of individual executions, including the calculation of an optimal order.

38.8.1  General Approach

The approach depicted below is based on the possibility to apply information about the name, the result and a free-form comment to each execution (stored in .csexe files). See Chapter 32.1 for the details.

As an example we’ll use Squish for Qt’s addressbook on a Unix-based system and a JavaScript test script.

  1. First we’ll initialize the execution data with the name of the Squish test case that is being run.
    function main()
    {
        var currentAUT = currentApplicationContext();
        var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
        var testCase = squishinfo.testCase;
        var testExecutionName = testCase.substr(testCase.lastIndexOf('/') + 1);
        var file = File.open(execution, "a");
        file.write("*" + testExecutionName + "\n");
        file.close();
        var ctx = startApplication("addressbook");
    ...
  2. Insert the main test script at this point
  3. After the main test script we’ll log the result of the test for the coverage tool:
    ...
        // wait until AUT shutdown
        while (ctx.isRunning) {
            snooze(1); // increase time if not enough to dump coverage data
        }

        // test result summary and status
        var positive = test.resultCount("passes");
        var negative = test.resultCount("fails") + test.resultCount("errors") + test.resultCount("fatals");
        var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
        var status = negative == 0 ? "PASSED" : "FAILED";
        var file = File.open(execution, "a");
        file.write("<html><body>" + msg + "</body></html>\n");
        file.write("!" + status + "\n")
        file.close();
    }

When you execute the scripts containing these steps, the Squish CocoExecution Report loads with the test case name, status and execution summary in the execution details and comments.

38.8.2  Simplified for Reuse

  1. Create a file called squishCocoLogging.js in Test Suite Resources with the following functions:
    function getExecutionPath() {
        var currentAUT = currentApplicationContext();
        var execution = currentAUT.cwd + "\\" + currentAUT.name + ".exe.csexe"
        return execution;
    }

    function logTestNameToCocoReport(currentTestCase, execution) {
        var testExecutionName = currentTestCase.substr(currentTestCase.lastIndexOf('\\') + 1);
        var file = File.open(execution, "a");
        file.write("*" + testExecutionName + "\n");
        file.close();
    }

    function logTestResultsToCocoReport(testInfo, execution){

        var currentAUT = currentApplicationContext();

        // wait until AUT shuts down
        while (currentAUT.isRunning)
          snooze(5);

        // collect test result summary and status
        var positive = testInfo.resultCount("passes");
        var negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") +
            testInfo.resultCount("fatals");
        var msg = "TEST RESULTS - Passed: " + positive +  " | " + "Failed/Errored/Fatal: " + negative;
        var status = negative == 0 ? "PASSED" : "FAILED";

        // output results and status to Coco execution report file
        var file = File.open(execution, "a");
        file.write("<html><body>" + msg + "</body></html>\n");
        file.write("!" + status + "\n")
        file.close();
    }

    A Python version of this code is:

    import re

    def getExecutionPath():
        currentAUT = currentApplicationContext()
        execution = "%(currAUTPath)s\\%(currAUTName)s.exe.csexe" % {"currAUTPath" : currentAUT.cwd, "currAUTName" : currentAUT.name}
        return execution

    def logTestNameToCocoReport(currentTestCase, execution):
        testExecutionName = re.search(r'[^\\]\w*$', currentTestCase)
        testExecutionName = testExecutionName.group(0)
        file = open(execution, "a")
        file.write("*" + testExecutionName + "\n")
        file.close()

    def logTestResultsToCocoReport(testInfo, execution):
        currentAUT = currentApplicationContext()
        # wait until AUT shuts down
        while (currentAUT.isRunning):
            snooze(5)

        # collect test result summary and status
        positive = testInfo.resultCount("passes")
        negative = testInfo.resultCount("fails") + testInfo.resultCount("errors") + testInfo.resultCount("fatals")
        msg = "TEST RESULTS - Passed: %(positive)s  |  Failed/Errored/Fatal: %(negative)s" % {'positive': positive, 'negative': negative}
        if negative == 0:
            status = "PASSED"
        else:
            status = "FAILED"

        # output results and status to Coco execution report file
        file = open(execution, "a")
        file.write("<html><body>" + msg + "</body></html>\n")
        file.write("!" + status + "\n")
        file.close()
  2. Add the following function calls after startApplication() in the main test script:
    execution = getExecutionPath();

    logTestNameToCocoReport(squishinfo.testCase);

    In Python:

    execution = getExecutionPath()

    logTestNameToCocoReport(squishinfo.testCase)
  3. At the end of your script, after closing the AUT (for example with steps clicking File > Exit), call the following function:
    logTestResultsToCocoReport(test);

    In Python:

    logTestResultsToCocoReport(test)
  4. In the event your AUT closes unexpectedly, or a script error occurs, incorporating a try, catch, finally ensures your results still output to the Coco report file.

Your main test script should be similar to the following:

source(findFile("scripts","squishCocoLogging.JS"))

function main()
{
    startApplication("addressbook");
    execution = getExecutionPath();
    logTestNameToCocoReport(squishinfo.testCase, execution);

    try {
        // body of script
    }
    catch(e) {
       test.fail('An unexpected error occurred', e.message)
    }
    finally {
       logTestResultsToCocoReport(test, execution)
    }
}

Python version:

source(findFile("scripts","squishCocoLogging.py"))

def main():
    startApplication("addressbook")
    execution = getExecutionPath()
    logTestNameToCocoReport(squishinfo.testCase, execution)

    try:
        try:
            # body of script
        except Exception, e:
            test.fail("test failed: ", e)
    finally:
        logTestResultsToCocoReport(test,execution)

Chapter 39  Atlassian Bamboo integration

39.1  Introduction

Atlassian Bamboo is a continuous integration server used to build, test and release software. The Squish Coco plug-in provides seamless Squish Coco integration with Bamboo. Users can add one or more Squish Coco tasks to a Bamboo Job. Each Squish Coco Task can perform the following actions:

  1. Import an execution report into an instrumentation data base
  2. Merge several instrumentations databases together
  3. Generate code coverage report

User can choose only one action to be performed, two actions or all three actions. If more than one action is chosen then they are executed in mentioned above order.

pictures/manual.tmp051.png
Figure 39.1: Squish Coco Task configuration overview

After executions finish, Artifacts for Squish Coco (like Coverage Report) can be stored to allow efficient defect troubleshooting.

39.2  Agent configuration

An Agent is a service that executes Bamboo builds, tests and deployments. The Agent Capabilities setting allows users to specify which agents are capable of executing Squish Coco jobs. In order to execute it , we must first install Squish Coco at Agent host. Next, we need to setup Agent-specific capability called ’coco’ with Squish Coco installation directory as a Value. (go to Bamboo administration|BUILD RESOURCES|Agents|AgentName|Agent-specific capabilities). Later on, during Job configuration, we need to set ’coco’ capability existence as a Job requirement. This way, Squish Coco jobs will be executed only on Agents where Squish Coco in installed.

39.3  Importing an execution report

This action calls cmcsexeimport to import execution report (CSExe file) into an instrumentation data base (CSMes file). User need to provide absolute path or relative path (from Bamboo build working directory) for both CSExe file and CSMes file and execution title. When this action is activated, the following command will be executed on Agent host:

cmcsexeimport --debug --title=<Title> -m <CSMesFile> -e <CSExeFile>

39.4  Merging instrumentation databases

This action calls cmmerge to merge several instrumentation databases (CSMes files) together. We need to provide CSMes output file full or relative path name that will be generate as a result of merge. If we provide relative path (or just filename), then CSMes output file will be generated in Bamboo working directory or relative sub directory. CSMes input files need to be specified, multiple files can be used separated by comma. When this action is activated, the following command will be executed on Agent host:

cmmerge --verbose -o <CSMesOutputFile> <CSMesInputFile1> ... <CSMesInputFileN>

Additionally user can choose option to ’merge only instrumentations and executions present in the reference file’. This option is useful for importing CSMes files with Unit Tests. When activated, used need to provide the name of CSMes reference file. This will cause that above cmmerge command will be executed with additional ’-i’ option.

cmmerge --verbose -o <CSMesOutputFile> -i <CSMesRefFile> <CSMesInputFile1> ... <CSMesInputFileN>

39.5  Generating coverage report

The last action calls cmreport to generate HTML Report. We need to provide CSMes file name (either absolute or relative path). CSMes file shall contain previously imported execution report (i.e. by using first action offered by plugin).

cmreport --title=<BambooJobName> -m <CSMesFile> --debug --html=coco/report.html

Generated report will be stored in coco subdirectory, therefore we can define Artefact with ’coco/**’ as a Copy pattern. This allows us to view the report directly in Bamboo after job execution. The report title is build using Bamboo Job name.

39.6  Use Case Examples

39.6.1  Coverage Report from single execution

We need to activate first action ’Import an execution report into an instrumentation data base’ to import execution report (CSEXE file) into an instrumentation data base (CSMES). Finally, we need to activate third action ’Generate code coverage report’. CSMES file defined for this action is the same as for first action.

pictures/manual.tmp052.png
Figure 39.2: Squish Coco Task configuration for example addressbook application

39.6.2  Coverage Report for Unit Tests

In this scenario we would like to generate coverage report for unit tests execution. To achieve this scenario in Bamboo we need to use Squish Coco task twice in one job.

After the build we have two CSMes files: one for our application and second for Unit Tests. We need to merge those instrumentation data bases where CSMes file for our application will be a reference during the merge. After the merge new CSMes output file is created.

pictures/manual.tmp053.png
Figure 39.3: Squish Coco Task configuration for merging Unit Tests executions

Using second Squish Coco task execution report from Unit Test run need to be imported into CSMes file (which was created in previous task - CSMes output file). Last action ’Generate code coverage report’ is used to generate HTML coverage report.

pictures/manual.tmp054.png
Figure 39.4: Squish Coco Task configuration for importing execution and generating report

Chapter 40  Jenkins CI integration

Jenkins CI is a popular continuous integration tool. Squish Coco can be run under Jenkins, and the changes of the coverage over time displayed by a plugin. This chapter describes the necessary setup.

40.1  Prerequisites

40.1.1  Getting a license

The Jenkins process runs under an own account, usually called “jenkins”. When therefore Squish Coco runs under Jenkins CI with a node-locked license, it needs a specific license for its account.

How to get this license is described in the following section; the section after that one describes the use of a license server.

Installing a node-locked license

In order to generate the license, the Squish Coco license tools need to be run from the “jenkins” account. There are two ways to do it, depending on whether the Jenkins server has an Internet connection.

Using a license server

For the use of Squish Coco with a license server, one has to create a new Jenkins CI project, which only runs (under Microsoft® Windows) the following command:

"Windows Coco\cocolic" --license-server=⟨host⟩:⟨port

where ⟨host⟩ and ⟨port⟩ refer to the license server (see Chapter 26). The colon and the port can be omitted if the default port is used.

Under Linux and macOS, this becomes

cocobin⟩/cocolic --license-server=⟨host⟩:⟨port

Then the license is installed.

40.1.2  Getting an EMMA plugin

Coco uses the EMMA-XML format to communicate the coverage results to Jenkins. Jenkins therefore needs a plugin to display EMMA data. We use here the plugin at https://wiki.jenkins-ci.org/display/JENKINS/Emma+Plugin. Install it in Jenkins with the Jenkins plugin manager.

40.2  Adaption of the project to Jenkins

We will assume here that you already compile and run your project under Jenkins and that you can also generate coverage information, but not yet under Jenkins.

With the license installed, code coverage should now work under Jenkins.

The build process must now be configured in such a way that a single .csmes file is generated that contains all the execution data. (This is done with cmcsexeimport.)

Now extend your build process such that at the end a report im EMMA-XML format is generated. This is done by calling

"Windows Coco\cmreport" -m project.csmes --emma=report.xml

where project.csmes must be replaced with the name of your results file.

We then use the plugin to display the data that were written to the file report.xml. To do this, select the configuration section of your Jenkins project and add a post-build action. Since the plugin is installed, the "Add post-build action" menu contains an entry "Record Emma coverage report". Select it, and in the field "Folders or files containing Emma XML reports" enter "report.xml".

Then after the next build, a coverage report will be generated.

pictures/manual.tmp055.png
Figure 40.1: Generating a coverage report from report.html with Jenkins CI.
email squish@froglogic.com