17.21. How to Create and Use Verification Points

17.21.1. How to Create and Use Object Property Verifications
17.21.2. How to Create and Use Table Verifications
17.21.3. How to Do Screenshot Verifications
17.21.4. How to Create and Use Verification Points in Test Scripts

Verification points are a central feature of automated testing. To create a verification point, we must first drive the AUT so that it reaches the state we want to verify, and then we must check that the AUT behaves as it should and has the correct state at the point we have reached. This is done by using verification points—these verify that object property values or screenshots or other testable aspects of the AUT are what they are expected to be.

Verification points can be created purely in code using the Squish API's Verification Functions (Section 18.3.8), or using the Squish IDE's point & click interface with the Verification Point Creator view (Section 20.2.19), or using a mixture of both. We will cover both approaches in this section.

17.21.1. How to Create and Use Object Property Verifications

The most common type of verification is to compare the value of one of an object's properties with an expected value. For example, after some text has been typed in, we could compare the line editor's text property's value with the text that was entered using one of Squish's test functions, such as test.compare and test.verify.

Verification points like this can easily be inserted into a test script using Squish IDE's point & click interface. A detailed step-by-step description on how to insert an object property verification point can be found in the tutorial. (See Tutorial: Starting to Test Qt Applications (Chapter 4), Tutorial: Starting to Test Java™ AWT/Swing Applications (Chapter 5), Tutorial: Starting to Test Web Applications (Chapter 8), and so on, depending on which GUI toolkit you are interested in.)

Object property comparisons can be inserted for all the properties that Squish recognizes for all the AUT objects that Squish detects. (For example, for Qt applications this includes all QObject based objects, which in turn includes all built-in and custom widgets since they all derive from QWidget which itself inherits QObject.)

Some objects are containers for other objects. For example, tree, table, and list widgets, as well as menus (with their menu items and submenus). Squish provides access to both container objects and to the objects they contain. (See for example, How to Test Items in Item Views, Item Widgets, and Models (Qt 4) (Section 17.2.6.3), or How to Test JList, JTable, and JTree widgets (Java—AWT/Swing) (Section 17.4.6.1.2), and so on, depending on which GUI toolkit you are interested in.)

17.21.2. How to Create and Use Table Verifications

From Squish 4.2 it is possible to verify entire tables (and grids and similar widgets) using a table verification point rather than having to do so manually in code. This feature is supported for Qt, Mac OS X, Java™, and Windows applications

If you are using an earlier version of Squish than 4.2, or a toolkit for which there is no table verification support, it is still possible to achieve the same effect using hand written code. (For examples of such code see How to Test Table Widgets and Use External Data Files (Qt 4) (Section 17.2.6.4), Section 17.4.6.1.2.2, “How to Test JTable”, or Section 17.4.6.2.2.2, “How to Test Table”.)

The steps for creating a table verification point are very similar to those needed to create object property or screenshot verifications.

If you are recording a new test script, whenever you want a table verification point, click the Control Bar Window (Section 20.1.3)'s Insert Object Properties Verification Point toolbar button ().

If you have an existing test script and want to include a table verification point, put a breakpoint on the line after where you want the verification to take place. (Use several breakpoints if you want to insert several table verifications.) Now replay the test: it will stop at the first breakpoint.

Once the recording or playback is stopped you can insert a new table verification point. To do this, check the checkbox for the table in the Application Objects view (Section 20.2.1) to make the Verification Point Creator view (Section 20.2.19) appear. Change the verification from “Scriptified VP” (the default) to “Table VP”. Now recheck the table's checkbox and then click the Insert button. The recording or playback will now continue. You can do this as often as you need—for example to check the table at different points during the test.

If a table verification fails, a Fail entry will be added to the test results. Invoke this entry's context menu (e.g., right-click it on many platforms) and choose the View Differences menu item to see the differences.

By default table verification does an exact textual match on every single cell—but if this fails, Squish tries to do a regular expression match on the cell's contents. This is useful because in some cases, a cell's contents might vary every run (e.g., a column of timestamp values). To allow for such cases it is possible to replace a cell's contents with a regular expression. For example, using the text .* in a cell will mean that the cell will match any text (including no text at all). To do this click the table verification point in the Test Suites view (Section 20.2.16)'s Test Case Resources list's VP tab to show the table in an editable view. Then click a cell and replace its text with a regular expression, repeating this for as many cells as necessary. (See also Regular Expression (regex) Matching with ~= (Section 19.9.4).)

17.21.3. How to Do Screenshot Verifications

This section discusses how to create and work with screenshot verification points.

17.21.3.1. How to Create and Use Screenshot Verifications

By far the most common kind of verifications are object property verifications, but in some situations it is useful to be able to visually compare how a widget (or group of widgets) appears with an image of how we expect them to appear. For this to work we must provide Squish with an image (e.g., a screenshot) that shows the widget(s) as they should look, so that Squish has something to compare with.

Although Squish makes it very easy to set up screenshot verifications, they are not always as useful or convenient as we might expect, and in most cases using object property verifications to compare values is the best approach. Screenshot verifications essentially work by comparing pixels, but this can very easily lead to incorrectly failing verifications. For example, if the test is run on a machine that is different from the one on which the original screenshot was taken, the screenshots taken during the test may differ if the test is run on a different operating system, or on a machine with a screen that has a different resolution. And even if the tests are run on the same machine, if the fonts or theme are changed the screenshots will differ. However, if we are careful about these issues, there are situations where screenshot verifications make sense.

One example where screenshot verifications are useful is when we need to verify that a graph or diagram has been drawn correctly.

It is just as easy to insert a screenshot verification as it is to insert an object property verification—in fact, the steps are almost the same. (See, for example, Inserting Additional Verification Points (Section 4.4).) One way is to set a breakpoint and run the test until the breakpoint is reached; another way is to invoke the Insert Screenshot Verification Point action (Section 20.1.3.3) during test recording. In either case, at this point the Application Objects view (Section 20.2.1) is shown. To verify a screenshot of a particular object (or of the whole window), first, check the object's check box. This will make the Verification Point Creator view (Section 20.2.19) appear. (If the object you're interested in isn't listed, interact with the AUT until the object you want is shown and either refresh the Application Objects view (Section 20.2.1) or use the object picker to select the object.)

At this point the verification has not been inserted! Choose “Screenshot VP” in the combobox and click the Insert button. If you started from an existing test and stopped at a breakpoint, either continue the test run or terminate it; and if you are inserting verifications during recording, simply continue recording and inserting until you are done. For each inserted verification there will be a call to the test.vp function—at the breakpoint, or at the point in recording where the insertion was made.

The Squish IDE showing a screenshot verification that is about to be added

When we insert the verification point any screenshots that have been taken are saved along with any object property values, as part of the expected results. Later on, when the test script is replayed, whenever Squish reaches a screenshot verification point it takes a screenshot of the relevant widget and compares this with the corresponding screenshot that it originally took, and reports success or failure for the verification point depending on whether the images are the same or not.

If a screenshot verification fails, in addition to recording the failure in the log, Squish also saves the new (and different) screenshot for later inspection. The Squish IDE also has a facility that allows us to view the differences between the expected image and the actual image by right clicking the failure message in the test log view to pop up the context menu, and choosing the View Screenshots Diff option.

Squish can show the differences between screenshots using a number of different techniques and algorithms—for example, subtraction of the images, a side-by-side view, and an animation where the expected and actual images are displayed quickly one after the other in a loop. If the actual image is correct (perhaps because of a change to the AUT that has made the original screenshot out of date), we can set the actual image as the new expected result.

17.21.3.2. Image Masks

Squish provides a tool which allows us to make screenshot verifications more robust. This is done by using image masks. These allow us to specify particular areas in the screenshot which should be ignored or respected in the comparison.

Image masks can be created and modified for a screenshot in the Squish IDE's verification point editor. The editor allows us to insert, modify and remove positive and negative masks.

17.21.4. How to Create and Use Verification Points in Test Scripts

Although it is possible to insert verification points through the Squish IDE without having to write a single line of code, it is also possible to add verifications directly in code. This is especially useful when a verification point needs to be more flexible than can be achieved using the point & click approach—for example, if we want to iterate over every item in a list view to verify each one.

There is no harm in starting by creating verification points using the Squish IDE's point & click interface, especially since it is easy to convert them into pure script code.

When a verification point is created using point & click, there are two possible ways that the verification can be recorded. The default (and recommended) way is to insert using the “Scriptified Properties VP” option in the Verification Point Creator view (Section 20.2.19)'s combobox. This will insert the verification using Squish's API, for example, using the test.compare function. An alternative is to use the “Properties VP” option. This will insert a call to the test.vp function with the details of what to verify saved in an external file. It is clearer and easier to maintain tests that use explicit verification functions such as test.compare which is why they are inserted by default.

To convert a call to the test.vp function into an explicit verification, click on the call to the test.vp function and invoke the context menu. Choose the Scriptify Verification Point menu option (this option does not appear for screenshot verifications). This will pop up the Refactoring wizard which shows “before” and “after” views of the test's code. If you are happy with the proposed change click the Finish button and the change will be applied.

For example, if we created a (non-screenshot) verification point through the Squish IDE—for example, to check the text of a button—and it was inserted into our test script like this:

test.vp("VP1")

If we then scriptified the verification point, it would be replaced with code something like this:

waitFor("object.exists(':Add_QPushButton')")
test.compare(findObject(":Add_QPushButton").text, "Add")

Naturally the actual verification code will match whatever the point & click verification point checked, and of course the resultant code will be in whichever scripting language is used for the test.

Note that for hand written verifications rather than using the waitFor function, we normally use the waitForObject function. So if we were doing this verification entirely by hand we would write it as a single line of code—for example:

test.compare(waitForObject(":Add_QPushButton").text, "Add")

For more details about the script API for verification test statements, see How to Use Test Statements (Section 17.8).

Here is a small example which shows a dynamic verification point that iterates over all the items in a Qt list widget—whose contents vary from run to run—and checks that no item's text is empty. Such a verification point cannot be created using the Squish IDE since the number of items is unknown and may be different on each test run. Such flexibility can only be achieved by using a scripting language:

Python
def main():
    listWidget = waitForObject(":Item Views_QListWidget")
    numberEmpty = 0
    for row in range(listWidget.count):
        item = listWidget.item(row)
        if item.text.isEmpty():
            numberEmpty += 1
    if numberEmpty:
        test.fail("Out of %d list items, %d were empty" % (
            listWidget.count, numberEmpty))
    else:
        test.passes("Verified %d non-empty list items" % (
            listWidget.count))
JavaScript
function main()
{
    var listWidget = waitForObject(":Item Views_QListWidget");
    var numberEmpty = 0;
    for (var row = 0; row < listWidget.count; ++row) {
        var item = listWidget.item(row);
        if (item.text.isEmpty())
            ++numberEmpty;
    }
    if (numberEmpty)
        test.fail("Out of " + listWidget.count + " list items, " +
            numberEmpty + " were empty");
    else
        test.pass("Verified " + listWidget.count +
            " non-empty list items");
}
Perl
sub main
{
    my $listWidget = waitForObject(":Item Views_QListWidget");
    my $numberEmpty = 0;
    for (my $row = 0; $row < $listWidget->count; ++$row) {
        my $item = $listWidget->item($row);
        if ($item->text->isEmpty()) {
            ++$numberEmpty;
        }
    }
    if (numberEmpty) {
        test::fail("Out of " . listWidget.count .
            " list items, $numberEmpty were empty");
    }
    else {
        test::pass("Verified " . listWidget.count .
            " non-empty list items");
    }
}
Ruby
# encoding: UTF-8
require 'squish'
include Squish

def main
    listWidget = waitForObject(":Item Views_QListWidget")
    numberEmpty = 0
    for row in 0...listWidget.count
        item = listWidget.item(row)
        if item.text.isEmpty()
            numberEmpty += 1
        end
    if numberEmpty != 0
        Test.fail("Out of #{listWidget.count} list items, " +
            "#{numberEmpty} were empty")
    else
        Test.pass("Verified #{listWidget.count} non-empty list items")
    end
end
Tcl
proc main {} {
    set listWidget [waitForObject ":Item Views_QListWidget"]
    set numberEmpty 0
    set count [property get $listWidget count]
    for {set row 0} {$row < $count} {incr row} {
        set item [invoke $listWidget item $row]
        if {[invoke [property get $item text] isEmpty]} {
            incr numberEmpty
        }
    }
    if {numberEmpty != 0} {
        test fail "Out of $count list items, $numberEmpty were empty"
    } else {
        test pass "Verified $count non-empty list items"
    }
}

The test obtains a reference to the list widget, then iterates over every item it contains, keeping a count of those that have empty texts. If any are empty the test.fail function is called; otherwise the test.pass function is called.

For examples that show how to iterate over all the items in Qt's item-based convenience widgets and for the models that Qt's views use, see How to Test Items in Item Views, Item Widgets, and Models (Qt 4) (Section 17.2.6.3). There are similar sections for Java tests, for example, How to Test JList, JTable, and JTree widgets (Java—AWT/Swing) (Section 17.4.6.1.2) and How to Test List, Table, and Tree widgets (Java™/SWT) (Section 17.4.6.2.2), and also for the other GUI toolkits that Squish supports.