5.24. How to Do Image-Based Testing

Table of Contents

5.24.1. Start a New Game
5.24.2. Move the Pawn
5.24.3. The Recorded Script
5.24.4. Tolerant image search
5.24.5. Verifications by Image-Search

This section explains the steps necessary to create test scripts using image-based look-ups. Retrieval of objects based on their screen appearance is a complementary approach to the property-based identification described in How to Identify and Access Objects (Section 5.1).

As a sample AUT, we will use Chess 2012, a game developed by Filip Höfer. The application employs several classic controls like push buttons and menus as can be seen in the screenshot below. Normally, it is recommended to access those controls through multi-property names like {type='Button' text='Start new game'}. For the sake of this how-to we will restrict ourselves to image-based object searches for a short test case.

Sample Application - Chess Game

We will automate a minimal sequence of steps. Granted, not the best opening but we will not go as far as the end game anyway.

  1. Start a new game

  2. Move the white pawn from square A2 to square A4.

5.24.1. Start a New Game

While we would normally automate the push of the Start new game button through a property-based object access (mouseClick(waitForObject(":Start_Button"))) we will now use the pixel representation of that button.

  1. Record () the Test Case.

  2. Invoke the Insert > mouseClick(<Image>) action (Section 8.1.3.6) ()

  3. Wait for the IDE to grab the screen content and move the red selection rectangle around the Start new game button label.

  4. Enter a meaningful file name like start_game.

  5. Press the Insert mouseClick button. This will a) perform the click on the button and b) queue the script statement mouseClick(waitForImage("start_game")) for insertion at the end of recording.

Start New Game

5.24.2. Move the Pawn

To start the move of pawn we will record a click on square C2. A click on square C4 will perform the move. Once we activate the Insert > mouseClick(<Image>) action (Section 8.1.3.6) () we get to choose previously recorded images:

Image Reuse or Creation

The image of the Start new game is not applicable for reuse at this point. So we twice use New Image... to record images for the two squares using the same steps as above.

The first image cut out from the chessboard will be the white pawn on square C2:

white_pawn.png

At this point there are (still) eight such pawns on the board. And four of them are even placed on same white background that we have captured. Upon accepting the selection Squish will search the desktop screenshot ,find all positions matching the selection and compute the occurrence index. That index will be recorded as a part of the waitForImage call. Therefore, it is important to select the interesting image even if there are other identical occurrences of it on the screen.

The next image to select is the white square C4. Recording search images that consist only of flat background is not recommended. In order to avoid that, the selected image should include margins of neighboring squares.

white_square.png

As before, other empty white squares will be considered, and the occurrence index recorded as the occurrence parameter to the waitForImage function.

The next recorded move is a pawn going from E2 to E4. We activate the Insert > mouseClick(<Image>) action (Section 8.1.3.6) () , and select the previously created white_pawn file. Since there are multiple pawns still visible on the board, Squish will open the Image Search Preview dialog (Section 8.3.4) with several matches marked with red frame.

Selection of Image Occurrence

This time the occurrence index recorded in the test script will be based on the selected image.

The same steps can be repeated to use the chess_square.png file to click on the square E4.

5.24.3. The Recorded Script

Once the image-based action insertions are done, we press the Stop Recording action () in the Control Bar. The generated script will look like this (or equivalent in your chosen scripting language):

JavaScript
function main() {
    startApplication("chess2012");
    mouseClick(waitForImage("start_game"));
    mouseClick(waitForImage("white_pawn", { occurrence: 2 }));
    mouseClick(waitForImage("chess_square", { occurrence: 7 }));
    mouseClick(waitForImage("white_pawn", { occurrence: 3 }));
    mouseClick(waitForImage("chess_square", { occurrence: 4 }));
}

At this point we have automated a test without having access to internals of the chess application. The drawback of this low barrier is the reliance on a particular visual appearance of the application.

5.24.4. Tolerant image search

In case of small changes in the appearance of the application, Squish can introduce tolerance to the image search in order to find matches against previously captured images.

The Chess 2012 application that we used to record the test script scales it's contents according to the amount of available space. By maximizing the application window we can resize the entire chessboard.

JavaScript
function main() {
    startApplication("chess2012");
    mouseClick(waitForImage("start_game"));
    mouseClick(waitForImage("white_pawn"));
}

If the above test case is executed with the Chess 2012 application's window maximized, the image search will fail, and the IDE will show the Image Not Found dialog (Section 8.3.3).

Image Not Found Dialog

We could try to select the Attempt to fix image search errors automatically. checkbox and allow Squish to enable the tolerant image search mode and adjust its parameters. The automatic fix procedure is not perfect and risks relaxing the image search to include bad matches. Instead we will click the Adjust Search Parameters button, that opens the Image Search Preview dialog (Section 8.3.4). The dialog will attempt to lower the threshold until any matches are found. It displays the image search parameters and allows manipulating it and observe the search preview according to currently set values.

Image Search Preview

For now, the automatically computed values should be correct, and we can confirm it by clicking the Set as Defaults. This will set the default image search parameters to the values displayed above and continue the test script execution.

The default values of the image search parameters can be edited on the Image Search (Section 8.2.16.4) tab of the Settings view (Section 8.2.16).

5.24.5. Verifications by Image-Search

So far the sole purpose of image searches were a later interaction. Like a Insert > mouseClick(<Image>) action (Section 8.1.3.6). The mere existence (or absence) of an image can also be be used for the purpose of testing of the GUI state however. To answer questions like: Does a warning sign appear? Is a status icon red or green? Or (in the case of a test for our chess application): whose player's turn is it?

To verify that it's the white player's turn we will first cut out an image encompassing the text label It is white's turn. by recording e.g. a mouseClick on the label:

Selection of the whites_turn.png image

The recorded script statement will be:

JavaScript
mouseClick(waitForImage("whites_turn.png"));

To turn this into a check we'll drop the mouseClick call and reduce ourselves to verifying that the image search by itself succeeds:

JavaScript
try {
   waitForImage("whites_turn.png", {timeout: 1000})
   test.pass("It is white's turn");
} catch (e) {
   test.fail("It is not white's turn. Must be black's.")
}

Note how we reduced the maximum time we are willing to wait from 20 seconds to 1 second. This is to speed up the verification in case of a negative result. In case we do not need to wait for an update of the UI before performing the test we can simply use findImage. It will error out right away on failure to find the specified image.

Note that a failure to find a specific image does not rule out with 100% safety that the information is not displayed with a differing visual appearance. The absence of one image does also not always mean that another state of pixels is present for sure. It is therefore prudent to have the failing check for one image to be followed by the check for its expected alternative. In above case this would be the visual representation of the text It is black's turn.