This section explains the steps necessary to set up the running of automated tests and how to process the results. (See also, Automated Batch Testing (Section 16.2).)
We start by creating a script that processes Squish's test results (output in XML format), and which generates an HTML file containing the results in a form that a web browser can display. Then we create a script that automatically runs test suites and automatically converts the test results into HTML files.
One important part of automating test runs is to present the test results in an accessible format so that the testing team can immediately spot any problems that have occurred. Squish can save the results of tests in an XML format that is suitable for arbitrary processing. In this section we will present a Python script that converts results in Squish's XML output format to HTML.
Squish's use of XML means that test results can easily be processed into formats suitable for other tools to use. This makes it possible to integrate the results of Squish test runs into a test management system. The processing needed to transform Squish's XML format data works the same way whether the output is HTML or some other format, so even though this section shows the transformation from XML to HTML, the concepts and approaches that are illustrated can be used for any other kind of transformation that you need.
In order to convert the Squish results XML format, we must first understand its format; this is described in detail in the Processing Test Results (Section 16.2.3) section in the User Guide (Chapter 15).
To put things in concrete terms we will look at an XML test results file
produced by running an example suite_addressbook_py
test suite that contains two test cases. The test cases check whether
entering data into a simple addressbook AUT works
as expected.
To generate the XML report, we run the test suite using squishrunner—of course the squishserver must already be running:
squishrunner --testsuite suite_addressbook_py --reportgen xml2,/tmp/results.xml
The valid values for --reportgen depend on the
Squish version. Squish 3.x supports
xml (old XML format kept for backward compatibility and
not recommended), xml2 (recommended XML format for
Squish 3),
xls (Excel™ format), and stdout (plain text).
Squish 4 supports the same formats, and in addition
xml2.1 (recommended XML format for Squish 4) and
xmljunit (same output as JUnit tests; less informative than
Squish's native XML formats).
Since we specified xml2 format, we get an XML
results file similar to the one shown below:
<SquishReport version="2.0">
<test name="suite_csvtable">
<prolog time="2009-07-31T16:14:40+01:00"/>
<test name="tst_editing">
<prolog time="2009-07-31T16:14:49+01:00"/>
<verification line="193" name=""
file="squish/examples/qt4/csvtable/suite_py/tst_editing/test.py">
<result type="PASS" time="2009-07-31T16:15:42+01:00">
<description>Verified</description>
<description type="DETAILED">True expression</description>
</result>
<result type="PASS" time="2009-07-31T16:15:42+01:00">
<description>Verified</description>
<description type="DETAILED">True expression</description>
</result>
</verification>
<epilog time="2009-07-31T16:15:44+01:00"/>
</test>
<test name="tst_context_menu">
<!-- many lines elided -->
</test>
<epilog time="2009-07-31T16:16:11+01:00"/>
</test>
</SquishReport>
The xml2 XML format is described in The xml2 XML Report Format (Section 16.2.3.1).
The Python script to convert the XML file to HTML is included with
Squish:
examples/regressiontesting/squishxml2html.py. This
script will work with Python 2.4 or any later
2.x version. (There is also an older, deprecated
script called
examples/regressiontesting/xml2result2html.py which
produces somewhat different HTML output.)
The script reads the Squish XML results file specified on the command line and writes the results as HTML files.
squishxml2html.py --dir html -i *.xml
If the files in the current directory were
results1.xml and results2.xml,
then the squishxml2html.py would write
html/index.html,
html/results1.html, and
html/results2.html under this directory, and use
ISO 8601 format date/times. The
html/index.html file has links to the actual
results files.
As the example indicates, the
squishxml2html.py has various command line
switches, including -d (or --dir) to
specify a different output directory for the HTML files,
-i (or --iso) to have date/times
output in the ISO 8601 format rather than the locale-specific format
used by the Squish IDE, and -p (or
--preserve) to preserve the original message formatting
(newlines and so on).
![[Note]](images/note.png) | Note |
|---|---|
To provide some compatibility with the deprecated
xml2result2html.py script, the
squishxml2html.py script will write its HTML
output to |
index.html file produced by running
squishxml2html.py
Both the screenshots use the -i option (ISO format
date/times) rather than using the default locale-specific date/times.
Now that we have seen how to produce the results in XML format and how to run the Python script to convert them to HTML, we will look at a Python script that can be used to automate the process.
The next step is to automate running the tests and automatically call the squishxml2html.py script to convert the XML reports into HTML.
Essentially, we just need to define a list of test suites and hosts and then, using a loop, the test suites are run on all hosts using squishrunner. The resulting XML files are then automatically converted into HTML using the squishxml2html.py program. The file names for the HTML result contain the date, test suite and host to ensure that they are unique.
The squishruntests.py script provides the
automation we need. It read the information it requires from a simple
.ini format file, by default
runtests.ini in the same directory as the
application itself. (Another .ini file can be
specified on the command line using the -i or
--ini option). Here is an example
runtests.ini file—only the
SQUISHDIR and SUITES key–values are
mandatory, so the others can be omitted:
# Use ~ to stand for your HOME directory---on Windows or Unix
SQUISHDIR = ~/consultancy/froglogic/squish/bin
SUITES = ~/consultancy/froglogic/squish/examples/qt4/addressbook/suite_py \
~/consultancy/froglogic/squish/examples/qt4/suite_canvas \
~/consultancy/froglogic/squish/examples/qt4/suite_canvas_js
RESULTSDIR = ~/consultancy/froglogic/squish/results
HOSTS = 127.0.0.1
PRESERVE = 0
ISO = 1
The SQUISHDIR key–value is mandatory and must be set
to the path where the Squish executables are found.
The SUITES key–value is mandatory and must be set
to one or more space-separated suite paths (each path must contain a
suite.conf file).
The RESULTSDIR key–value is optional and should be set
to the path where the results XML files and HTML files must go. (This
directory will be created if it doesn't exist; it defaults to the
current directory).
The HOST key–value is optional and should be set
to one or more space-separated host names or IP addresses—every
test suite will be executed on every host. (It defaults to one host,
127.0.0.1, the local machine.)
The PRESERVE key–value is optional and defaults to 0
(don't preserve the formatting of messages). If set to 1, the format of
log messages and similar is preserved in the resultant HTML files.
The ISO key–value is optional and defaults to 0 (use
the locale-specific date/time format); set it to 1 to use ISO 8601
date/time format.
As the example file illustrates, long lines can be split over multiple
lines by escaping newlines. Also, ~ can be used to stand
for your home directory.
When running this script it is assumed that squishserver is running on all the specified hosts, and that the AUT paths are set up so that the AUTs run by the test suites can be found.
The last step would be to have the squishruntests.py
script run automatically—for example, once every night—to
ensure that no regressions have been introduced into the AUT. On Unix
this can be done by setting up a cron job which executes the script. On
Windows it might be possible to set up a Windows
Service to run the script regularly—contact froglogic support
for advice if you want to try this.
For detailed documentation about cron jobs or Windows Services, search for the relevant information on the Internet or contact froglogic's commercial support to assist you.
At the heart of the squishxml2html.py program
is its Squish XML results parser. This is implemented as a standard
xml.sax.handler.ContentHandler subclass that reimplements
just four methods—__init__,
startElement, characters, and
endElement. It should be straightforward to copy and
modify this code to perform your own custom parsing of reports in the
Squish xml2 format.
This section showed how to set up automated test runs using Squish's command line tools and two Python scripts. Of course a lot more can be done to customize how results are presented, statistics are calculated, etc., by parsing Squish's XML results and transforming the data in the ways that meet your needs.