5.25. How to Do Automated Batch Testing

Table of Contents

5.25.1. Processing Test Results
5.25.2. Automatically Running Tests
5.25.3. Conclusion

This section explains the steps necessary to set up the running of automated tests and how to process the results. (See also, Automated Batch Testing (Section 7.1).)

We start by creating a script that processes Squish's test results (output in XML format), and which generates an HTML file containing the results in a form that a web browser can display. Then we create a script that automatically runs test suites and automatically converts the test results into HTML files.

5.25.1. Processing Test Results

5.25.1.1. Generating HTML Test Results

Besides of the XML, JSON and other report types Squish is able to generate reports in much convenient for viewing format. In this section we will explain how to generate Squish's results in HTML format.

Squish's use of HTML means that test results can easily be processed and viewed in a web browser.

For demonstration we will look at an HTML results produced by running the example suite_py test suite we used before. The test case checks whether entering some data into a simple addressbook AUT works as expected.

To generate the HTML report, we run the test suite using squishrunner—of course the squishserver must already be running:

squishrunner --testsuite suite_addressbook_py --reportgen html,/tmp/results

As you problably noticed, we used --reportgen option again and provided html as a value. The syntax is the same as we used for other report types. Besides of mentioning the report type we also specified the results directory /tmp/results where the final data will end up. If the directory already exists and contains results from the previous executions, the new execution results will be appended to the existing ones and produce an aggregated report.

Since we specified html format, squishrunner will generate the HTML report in the specified directory. You can open the index.html results file in any web browser for viewing and analyzing.

Example Web Report generated by squishrunner run with --reportgen option

5.25.1.2. Generating XML Test Results

One important part of automating test runs is to present the test results in an accessible format so that the testing team can immediately spot any problems that have occurred. Squish can save the results of tests in an XML format that is suitable for arbitrary processing. In this section we will present a Python script that converts results in Squish's XML output format to HTML.

Squish's use of XML means that test results can easily be processed into formats suitable for other tools to use. This makes it possible to integrate the results of Squish test runs into a test management system. The processing needed to transform Squish's XML format data works the same way whether the output is HTML or some other format, so even though this section shows the transformation from XML to HTML, the concepts and approaches that are illustrated can be used for any other kind of transformation that you need.

In order to convert the Squish results XML format, we must first understand its format; this is described in detail in the Processing Test Results (Section 7.1.3) section in the User Guide (Chapter 5).

To put things in concrete terms we will look at an XML test results file produced by running an example suite_py test suite that contains one test case. The test case checks whether entering some data into a simple addressbook AUT works as expected.

To generate the XML report, we run the test suite using squishrunner—of course the squishserver must already be running:

squishrunner --testsuite suite_addressbook_py --reportgen xml3,/tmp/results

The valid values for --reportgen depend on the Squish version, and are listed in squishrunner --reportgen: Generating Reports (Section 7.4.3.5).

Since we specified xml3 format, we get an XML results file similar to the one shown below:

<?xml version="1.0" encoding="UTF-8"?>
<SquishReport version="3" xmlns="http://www.froglogic.com/resources/schemas/xml3">
    <test type="testsuite">
        <prolog time="2015-06-19T11:22:27+02:00">
            <name><![CDATA[suite_test]]>%lt;/name>
            <location>
                <uri><![CDATA[file:///D:/downloads/simple]]>%lt;/uri>
            </location>
        </prolog>
        <test type="testcase">
            <prolog time="2015-06-19T11:22:27+02:00">
                <name><![CDATA[tst_case1]]></name>
                <location>
                    <uri><![CDATA[x-testsuite:/tst_case1]]>%lt;/uri>
                </location>
            </prolog>
            <verification>
                <location>
                    <uri><![CDATA[x-testcase:/test.py]]></uri>
                    <lineNo><![CDATA[2]]></lineNo>
                </location>
                <scriptedVerificationResult time="2015-06-19T11:22:27+02:00" type="PASS">
                    <scriptedLocation>
                        <uri><![CDATA[x-testcase:/test.py]]>%lt;/uri>
                        <lineNo><![CDATA[2]]></lineNo>
                    </scriptedLocation>
                    <text><![CDATA[Verified]]></text>
                    <detail><![CDATA[True expression]]></detail>
                </scriptedVerificationResult>
            </verification>
            <verification>
                <location>
                    <uri><![CDATA[x-testcase:/test.py]]></uri>
                    <lineNo><![CDATA[3]]></lineNo>
                </location>
                <scriptedVerificationResult time="2015-06-19T11:22:27+02:00" type="FAIL">
                    <scriptedLocation>
                        <uri><![CDATA[x-testcase:/test.py]]></uri>
                        <lineNo><![CDATA[3]]></lineNo>
                    </scriptedLocation>
                    <text><![CDATA[Comparison]]></text>
                    <detail><![CDATA['foo' and 'goo' are not equal]]></detail>
                </scriptedVerificationResult>
            </verification>
            <epilog time="2015-06-19T11:22:27+02:00"/>
        </test>
        <epilog time="2015-06-19T11:22:27+02:00"/>
    </test>
</SquishReport>

The xml3 XML format is described in The xml Report Format (Section 7.1.3.1).

5.25.2. Automatically Running Tests

The next step is to automate running the tests and automatically call the squishxml3html.py script to convert the XML reports into HTML.

Essentially, we just need to define a list of test suites and hosts and then, using a loop, the test suites are run on all hosts using squishrunner. The resulting XML files are then automatically converted into HTML using the squishxml3html.py program. The file names for the HTML result contain the date, test suite and host to ensure that they are unique.

The squishruntests.py script provides the automation we need. It reads the information it requires from a simple .ini format file, by default runtests.ini, in the same directory as the squishruntests.py script itself. (Another .ini file can be specified on the command line using the -i or --ini option). Here is an example runtests.ini file—only the SQUISHDIR and SUITES key–values are mandatory, so the others can be omitted:

# Use ~ to stand for your HOME directory---on Windows or Unix
SQUISHDIR = ~/opt/squish/bin
SUITES = ~/testing/myapp/version1/suite_main \
        ~/testing/myapp/version1.1/suite_main \
        ~/testing/myapp/version1.1/suite_extra
RESULTSDIR = ~/testing/myapp/results
HOSTS = 127.0.0.1
PRESERVE = 0
ISO = 1

The SQUISHDIR key–value is mandatory and must be set to the path where the Squish executables are found. The SUITES key–value is mandatory and must be set to one or more space-separated suite paths (each path must contain a suite.conf file). The RESULTSDIR key–value is optional and should be set to the path where the results XML files and HTML files must go. (This directory will be created if it doesn't exist; it defaults to the current directory). The HOSTS key–value is optional and should be set to one or more space-separated host names or IP addresses—every test suite will be executed on every host. (It defaults to one host, 127.0.0.1, the local machine and using Squish's default port number.) Each host name or IP may be followed by a colon and a port number, (e.g., 192.0.4.67:9812). The PRESERVE key–value is optional and defaults to 0 (don't preserve the formatting of messages). If set to 1, the format of log messages and similar is preserved in the resultant HTML files. The ISO key–value is optional and defaults to 0 (use the locale-specific date/time format); set it to 1 to use ISO 8601 date/time format.

As the example file illustrates, long lines can be split over multiple lines by escaping newlines. Also, ~ can be used to stand for your home directory.

When running this script it is assumed that squishserver is running on all the specified hosts, and that the AUT paths are set up so that the AUTs run by the test suites can be found.

The last step would be to have the squishruntests.py script run automatically—for example, once every night—to ensure that no regressions have been introduced into the AUT. On Unix this can be done by setting up a cron job which executes the script. (Since Windows Services don't support a display for running GUI applications, it is not possible to execute the squishserver as a Windows Service.)

For detailed documentation about cron jobs, search for the relevant information on the Internet or contact froglogic's commercial support to assist you.

5.25.3. Conclusion

At the heart of the squishxml3html.py program is its Squish XML results parser. This is implemented as a standard xml.sax.handler.ContentHandler subclass that reimplements just four methods—__init__, startElement, characters, and endElement. It should be straightforward to copy and modify this code to perform your own custom parsing of reports in the Squish xml2.1 format.

This section showed how to set up automated test runs using Squish's command line tools and two Python scripts. Of course a lot more can be done to customize how results are presented, statistics are calculated, etc., by parsing Squish's XML results and transforming the data in the ways that meet your needs—and since Squish outputs XML you can use any language you like for these purposes.