5.24. How to Do Image-Based Testing

Table of Contents

5.24.1. Start a New Game
5.24.2. Move the Pawn
5.24.3. The Recorded Script
5.24.4. Verifications by Image-Search

This section explains the steps necessary to create test scripts using image-based look-ups. Retrieval of objects based on their screen appearance is a complementary approach to the property-based identification described in How to Identify and Access Objects (Section 5.1).

As a sample AUT, we will use Chess 2012, a game developed by Filip Höfer. The application employs several classic controls like push buttons and menus as can be seen in the screenshot below. Normally, it is recommended to access those controls through multi-property names like {type='Button' text='Start new game'}. For the sake of this how-to we will restrict ourselves to image-based object searches for a short test case.

Sample Application - Chess Game

We will automate a minimal sequence of steps. Granted, not the best opening but we will not go as far as the end game anyway.

  1. Start a new game

  2. Move the white pawn from square A2 to square A4.

5.24.1. Start a New Game

While we would normally automate the push of the Start new game button through a property-based object access (mouseClick(waitForObject(":Start_Button"))) we will now use the pixel representation of that button.

  1. Start test case recording ().

  2. Invoke the Insert > mouseClick(<Image>) action (Section 8.1.3.6) ()

  3. Wait for the IDE to grab the screen content and move the red selection rectangle around the Start new game button label.

  4. Enter a meaningful file name like start_game.png.

  5. Press the Insert mouseClick button. This will a) perform the click on the button and b) queue the script statement mouseClick(waitForImage("start_game.png")) for insertion at the end of recording.

Start New Game

5.24.2. Move the Pawn

To start the move of pawn we will record a click on square A2. A click on square A4 will perform the move. Once we activate the Insert > mouseClick(<Image>) action (Section 8.1.3.6) () we get to choose previously recorded images:

Image Reuse or Creation

The image of the Start new game is not applicable for reuse at this point. So we twice use New Image... to record images for the two squares using the same steps as above.

The first image cut out from the chessboard will be the white pawn:

white_pawn.png

At this point there are (still) eight such pawns on the board. And four of them are even placed on same white background that we have captured. As the waitForImage function searches the screen from the left to the right (and top to the bottom) our pawn in column A will be found first.

To uniquely describe the target square A4 we will use a little trick and include the row label 4 into the cut out part of the image:

square_a4.png

We resort to including additional pixels to avoid matching any of the other empty white squares on the board. An alternative approach would be the specification of an occurrence to the waitForImage function.

5.24.3. The Recorded Script

Once the image-based action insertions are done, we press the Stop Recording action () in the Control Bar. The generated script will look like this (or equivalent in your chosen scripting language):

JavaScript
function main() {
    startApplication("chess2012");
    mouseClick(waitForImage("start_game.png"));
    mouseClick(waitForImage("white_pawn.png"));
    mouseClick(waitForImage("square_a4.png"));
}

At this point we have automated a test without having access to internals of the chess application. The drawback of this low barrier is the reliance on a particular visual appearence of the application. Future versions of Squish will be able to handle size and color differences, though.

5.24.4. Verifications by Image-Search

So far the sole purpose of image searches were a later interaction. Like a Insert > mouseClick(<Image>) action (Section 8.1.3.6). The mere existance (or absence) of an image can also be be used for the purpose of testing of the GUI state however. To answer questions like: Does a warning sign appear? Is a status icon red or green? Or (in the case of a test for our chess application): whose player's turn is it?

To verify that it's the white player's turn we will first cut out an image encompassing the text label It is white's turn. by recording e.g. a mouseClick on the label:

Selection of the whites_turn.png image

The recorded script statement will be:

JavaScript
mouseClick(waitForImage("whites_turn.png"));

To turn this into a check we'll drop the mouseClick call and reduce ourselves to verifying that the image search by itself succeeds:

JavaScript
try {
   waitForImage("whites_turn.png", {timeout: 1000})
   test.pass("It is white's turn");
} catch (e) {
   test.fail("It is not white's turn. Must be black's.")
}

Note how we reduced the maximum time we are willing to wait from 20 seconds to 1 second. This is to speed up the verification in case of a negative result. In case we do not need to wait for an update of the UI before performing the test we can simply use findImage. It will error out right away on failure to find the specified image.

Note that a failure to find a specific image does not rule out with 100% safety that the information is not displayed with a differing visual appearance. The absence of one image does also not always mean that another state of pixels is present for sure. It is therefore prudent to have the failing check for one image to be followed by the check for its expected alternative. In above case this would be the visual representation of the text It is black's turn.