Menu

Squish Coco

Code Coverage Measurement for Tcl, QML, C# and C/C++

Part V
QML Coverage

Chapter 13  Qt QML coverage

13.1  Overview

QML is a user interface markup language that is used in the Qt Framework. Squish Coco has an add-on that supports code coverage for QML. The add-on is delivered separately and requires an additional license.

It consists of a replacement for a Qt shared library file. By default the new library acts the same way as the version supplied by Qt, unless QML coverage is activated by setting an environment variable.

There is one exception from this rule. It concerns the QML disk cache, which interferes with correct instrumentation when switched on. In Qt 5.8 and later, where a disk cache exists, it is therefore always deactivated in the replacement library, even when instrumentation is not enabled.

13.2  Installation

The coverage add-on consists of a replacement of the Qt shared library Qt5Qml and is delivered in a ZIP archive. There exist several versions of the add-on, each for a different combination of Qt version and platform. To install it, one needs to download the right version of the archive, unpack it and put its content in place of the original library.

It is useful to keep a backup version of the original library files.

13.2.1  General remark

In the following sections we describe the installation for the most common case, where a Qt development kit has been installed in the system and an application is compiled to use it.

Please note that even if you have installed such a development kit, its library is not always used. Sometimes the application you want to trace has its own version of the Qtlibraries. Then you need to find the copy of the Qt5Qml library that it uses and replace this library with the froglogic version.

13.2.2  Finding the right version of the archive

The archives have names of the form “cocoqml-coco-version-qtqt-version-compiler-platform.zip”, with the following components:

coco-version
The version of Coco to which the add-on belongs. It has the form x.y.z, like e.g. 4.0.3.

All versions of Squish Coco with a release number later than that of ⟨coco-version⟩ can then read the coverage data that are generated by the add-on. However, since each Squish Coco release can read the data generated by earlier releases, it is not necessary to update the add-on when a newer version of Squish Coco is installed.

qt-version
The Qt version for which the library is compiled. It has the form a.b.c, like e.g. 5.8.0.

Note that a new version of the add-on is necessary whenever the patch level (the third number in the version string) of the Qt installation changes.

compiler
A string that identifies the compiler for which the Qt library is built. Examples are gcc_64 or msvc2013.
platform
Either linux, linux_64, mac or windows.

13.2.3  Linux installation

Under Linux, the archive contains a library file with a name like libQt5Qml.so.5.8.0. The numbers at the end are those of the Qt version.

To find the right version of the archive to download, you can usually use the command qmake that is used for the compilation of your project. If Qt is installed in the /opt/Qt/ directory, the full path to qmake might be /opt/Qt/5.8/gcc_64/bin/qmake. This means that the ⟨compiler⟩ is gcc_64 and the ⟨qt-version⟩ is a version number that begins with 5.8. To get the full version number, run qmake in the form

$ /opt/Qt/5.8/gcc_64/bin/qmake -query QT_VERSION
5.8.0

This then tells us that the full ⟨qt-version⟩ must be 5.8.0.

The correct package is then, for a 64-bit Linux system, “cocoqml-4.1.0-qt5.8.0-gcc_64-linux_64.zip”. Unpack it and copy the file libQt5Qml.so.5.8.0 contained in it to the directory /opt/Qt/5.8/gcc_64/lib.

Apart from libQt5Qml.so.5.8.0, there are also other files in the same directory that also begin with libQt5Qml.so. These files are only symbolic links and need not to be changed.

13.2.4  macOS installation

Under macOS, the archive contains a library in the form of a directory, with the name QtQml.framework/.

To find the right version of the archive to download, you can usually use the command qmake that is used for the compilation of your project. If Qt is installed in the /Users/someone/Qt/ directory, the full path to qmake might be /Users/someone/Qt/5.8/clang_64/bin/qmake. This means that the ⟨compiler⟩ is clang_64 and the ⟨qt-version⟩ is a version number that begins with 5.8. To get the full version number, run qmake in the form

$ /Users/someone/Qt/5.8/clang_64/bin/qmake -query QT_VERSION
5.8.0

This tells us that the full ⟨qt-version⟩ must be 5.8.0.

The correct package is then “cocoqml-4.1.0-qt5.8.0-clang_64-mac.zip”. Unpack it and copy the directory QtQml.framework/ contained in it to the directory /Users/someone/Qt/5.8/clang_64/lib/. (You need to rename the original QtQml.framework/ directory.)

13.2.5  Microsoft® Windows installation

Under Microsoft® Windows, the archive contains two Qt library files, Qt5Qml.dll and Qt5Qmld.dll, together with some more DLL files that are needed by the enhanced QML libraries.

To find the right version of the archive to download, you can usually use the command qmake that is used for the compilation of your project. If Qt is installed in the C:\Qt\ directory, the full path to qmake might be C:\Qt\5.8\msvc2013\bin\qmake. This means that the ⟨compiler⟩ is msvc2013 and the ⟨qt-version⟩ is a version number that begins with 5.8. To get the full version number, run qmake in the form

C:\Users\Someone>C:\Qt\5.8\msvc2013\bin\qmake -query QT_VERSION
5.8.0

This tells us that the full ⟨qt-version⟩ must be 5.8.0.

This then means that the correct package is “cocoqml-4.1.0-qt5.8.0-msvc2013-windows.zip”. Unpack it and copy the DLL files contained in it to the directory C:\Qt\5.8\msvc2013\bin\.

13.3  CocoQML Tutorial

The following tutorial provides examples how to use CocoQML. It also shows how to detect whether CocoQML is installed correctly.

The following examples are written for a UNIX® system. Microsoft® Windows is similar.

13.3.1  Compile a Qt application

We will use the calqlatr program as our example. It is part of the Qt distribution and can be found in the Examples/Qt-5.8/quick/demos/calqlatr/ subdirectory of your Qt installation.

The easiest way to set it up is to use the Qt Creator. Start the Qt Creator, select the "Welcome" mode and then "Examples". It is then possible to search for “calqlatr” and open the project.

Make sure that the compiler toolkit used for compiling the calqlatr project agrees with that of the CocoQML add-on. If you have e.g. downloaded and installed the package cocoqml-4.0.3-qt5.8.0-gcc_64-linux_64.zip, you need to select the Qt 5.8.0 GCC 64bit kit in the Qt Creator to compile it.

Compile the program and run it. Check whether it runs correctly.

13.3.2  Enable code coverage

Now you can enable code coverage by setting the COCOQML variable to 1. To do this, select "Projects" in the Qt Creator and add COCOQML=1 to the "Run Environment". Then run calqlatr again.

In the "Application Output" window you now should see lines like the following:

Starting /home/user/calqlatr/calqlatr...
CocoQML: Instrumentation database for 'calqlatr.qml' written to /home/user/calqlatr/calqlatr_qml.csmes
CocoQML: Instrumentation database for 'calculator.js' written to /home/user/calqlatr/calqlatr_qml.csmes
CocoQML: Instrumentation database for 'NumberPad.qml' written to /home/user/calqlatr/calqlatr_qml.csmes
CocoQML: Instrumentation database for 'Button.qml' written to /home/user/calqlatr/calqlatr_qml.csmes
CocoQML: Instrumentation database for 'Display.qml' written to /home/user/calqlatr/calqlatr_qml.csmes

If it does not, there is an error in the setup of CocoQML.

Now do some computations with the calqlatr, then close the program window. (Do not kill it from the Qt Creator, because then it cannot write the coverage data.) You should see an additional output similar to this:

CocoQML: Saved coverage information: /home/user/calqlatr/calqlatr_qml.csexe
/home/user/calqlatr/calqlatr exited with code 0

From this output we can see that the files calqlatr_qml.csmes and calqlatr_qml.csexe have been created in the directory /home/user/calqlatr. You can now open them with the CoverageBrowserand view the results.

13.3.3  Excluding source files from coverage

There is a common use case when some source files need to be excluded from instrumentation – maybe because they cause instrumentation errors or because coverage data is not needed for them. We will now show how this is done and exclude the file Button.qml from the code coverage measurements.

Source code exclusion is controlled by the environment variable COCOQML_BLACKLIST, the value of which must be the path to a text file. So we need to create he text file first.

Create a file with the name cocoqml.txt. It should contain one line of text,

Button.qml

Then add the variable COCOQML_BLACKLIST to the "Run Environment" section in the Qt Creator. Its value should be the full path to the cocoqml.txt file, like “/home/user/cocoqml.txt”.

Then the file Button.qml (and in fact all files with a name that contains the string “Button.qml”) will be excluded from instrumentation. Run the calqlatr program again. In the "Application Output" window you should now see a similar output as before, but with the Button.qml line missing. Similarly, the file Button.qml will be no longer be visible in the CoverageBrowser.

13.4  Resolving instrumentation problems

The following is a short recipe that tells what to do when the instrumented program does not run or behaves not the same way as the uninstrumented program.

  1. If logging was disabled, enable it by running the instrumented program with the environment variable COCOQML_VERBOSE set to 1. The output may contain hints about files that cause problems.
  2. If the output shows that a source file causes problems, you can exclude it from instrumentation with the COCOQML_BLACKLIST mechanism (see Chapter 13.3.3).
  3. If no error is logged but the instrumented program still behaves strange, you can try to find the problem file (or files) by selectively excluding files.

    The idea is to exclude selectively files to see whether they cause problems. In a small project, one can enter the name of a single source file into the blacklist file and then run the program again. If it now runs without problems, that file was the cause. Otherwise, repeat this procedure with another file, until the error is found.

    With larger projects, it is better to start with excluding whole subdirectories. If the name of a directory is entered in the COCOQML_BLACKLIST file, all files in that directory are excluded from instrumentation. If the program runs without problems with a directory excluded from instrumentation, the problem must be a file in this directory. You then need to repeat the test with its subdirectories or, if it is small enough, with its files. Otherwise, repeat the test with the other directories.

    This way, all files that cause instrumentation problems can usually be found and blacklisted.

  4. If there are still problems, mail the log file together with a description of the problem to froglogic support.

13.5  Environment Variables

When the Coco add-on is present, QML coverage is determined by environment variables. By default, instrumentation and coverage are switched off.

The following variables can be used to control code coverage:

COCOQML
Set this variable to 1 to enable the instrumentation of your QML source code. If the variable is not set or set to a different value, no instrumentation will be done: The Qt installation then behaves as usual.
COCOQML_OUTPUT
This variable contains the path to the directory in which the output files are generated. If the variable is not set, the current working directory of the program is used.

Setting this variable is especially useful if you do not have sufficient permissions in the automatically determined paths or want to gather all intermediate instrumentation databases in a single file.

COCOQML_CACHE
If this variable is set, CocoQML uses a cache to store the instrumentations of the source files. Its value must be the path to an empty directory which CocoQML uses to store the cache files.

With the cache enabled, CocoQML instruments a source file only if it cannot find it in the cache or if the cache contains a different version of the file. The new instrumented version of the file is then stored in the cache.

COCOQML_BLACKLIST
Set this variable to exclude certain paths from instrumentation. The value of the variable is the path to a text file which contains in each line a regular expression. If one of the regular expressions matches a file name, the file is excluded from the instrumentation.

Lines beginning with a “#” are ignored.

The following variables control various debug output generated by the extension, especially the log messages. All log messages are prefixed with “CocoQML: ” (without the quotes).

COCOQML_QUIET
Set this variable to 1 if you want to disable all messages generated by the add-on. Usually all messages will be printed to standard error, but if you rely on parsing standard error in your application you can use this to avoid interference.

This setting overrides the verbose mode, see below.

COCOQML_VERBOSE
Set this variable to 1 if you want to enable verbose mode. If the verbose mode is active, the CocoQML extension will print additional messages that might be helpful for debugging purposes. Usually you will not need to enable verbose mode.

This settings can be overridden by the quiet mode, see above.

COCOQML_LOGFILE
If this variable is set, the CocoQML add-on writes its log messages to a file. The value of COCOQML_LOGFILE must be an absolute or relative file path. A relative file path is interpreted relative to the working directory of the application when it starts. If the log file already exists, the new data is appended to it. If the path contains directories, they must already exist, otherwise no log file will be created.

Example:  Under Unix, one could set COCOQML_LOGFILE=/tmp/cocoqml.log. Since the directory /tmp/ exists in any normal Unix installation and is writable, all log messages are then written to the file /tmp/cocoqml.log

COCOQML_DUMPLOC
If this variable is set, the CocoQML add-on writes the instrumented versions of the QML files it instruments to the disk.

The value of COCOQML_DUMPLOC must be the path to a directory, either absolute or relative. A relative path is interpreted relative to the working directory at start time. If the directory does not exist, it is created.

email squish@froglogic.com