6. Creating a Test Activity

6.1. Introduction

To make the connection from the Start Test Action to the New_Account test action, deselect the New Action button by clicking on it. It will revert back to a plain background and the Test Activity diagram is now in Test Action flow connection mode, you will see that the cursor will become an arrow head rather than the cross in the New Action instantiation mode.

All Test Actions have flow nodes which are visible as white circles. Move your cursor to the green Start (setUp) Test Action, you will see that the white flow node for the connection will highlight in red when you hover over it. Click on the flow node and hold your left mouse button down. Drag your mouse, still holding down the left mouse button, to a flow node on New_Account test action. There are several flow nodes on each test action. Wait for the flow node to highlight in red before releasing your mouse button (this indicates that you are ‘over’ a flow node) and you should now see a flow arrow, a black dotted connection arrow from the Start Test Action to the New_Account test action. The arrow head and the direction of the flow of dashes when selected denotes the direction of test action flow from one action to another. To enable a Test Action to call itself, draw a flow arrow from one flow node to another flow node on the same test action.

Complete the flows as shown below, this creates a very simple Test Activity just using New_Account and Add_Particpant test actions that will ‘run’ to create 1 new account with up to 5 new participants.

6.2. Setting Seeds

When a Vitaq Test Activity is run the test generator randomly selects the next allowable test action in the test activity diagram, starting from the green Start (setUp) Test Action. Click on the green Start (setUp) Test Action to select it. You will notice that the Action Properties tab on the top right-hand side of the Vitaq Test Automation window automatically switches to the Start Test Action properties. The Start Test Actions properties are the seed control fields for Vitaq. By default, the starting seed for the Vitaq intelligent generator is set to 1. Whenever a different seed is used the Vitaq intelligent generator will potentially generate a different path through the test activity diagram. Vitaq is designed to be randomly stable, in other words if the same seed is used it will always produce the same path though the test activity. Seeds can be set as individual numbers, as a range of numbers e.g. ‘1-100’ or comma separated variable combinations such as ‘1,50-100,255,278’ where Vitaq will run the Test Activity with a seed value of 1, then seed values of 50, 51 and all seed numbers up to to 100 then seed number 255 and finally 278. You may want to do this because the auto-generated tests that were created by a specific seed may have uncovered a defect in the software under test and once fixed you may want to make sure Vitaq re-creates the same tests to test the fix.

Note

When running with the Edit->Preferences use AI radio button selected, Vitaq AI will automatically take control of setting seeds and running with new seeds to help the machine learning algorithms achieve full qa coverage.

Vitaq can also auto-increment the seed value from a starting seed number by clicking on the Auto-increment seed selector. This will increase the seed value by one each time you ‘run’ your Test Activity.

6.3. Vitaq output windows

At the bottom of the Vitaq window there are two tabs, Test Activity Log and Script Output. These tabs provide information on a test activity whilst it is being run. They provide useful information for test activity debugging and logging.

The two tabs are defined as:

Vitaq_logs

Tab	Description
Test Activity log	This displays the test id and seed value for each run. The test actions and test activity variable values Vitaq AI is choosing in each run as they are being executed. Then finally a log of the sequence of actions executed for that run.
Script Output	This contains all the test script output messages and Vitaq run control messages.

Note

The output windows do not keep a cache of the complete run, as this could become very large to render. A complete log of both output windows is saved on disk in your home directory in your docker container in /home/vitaq. They are named Vitaq_Script.log and Vitaq_Activity.log, each time VitaqAI is run in the Docker container theses logs have the logging data output of the run appended onto the appropriate log file.

Warning

When you stop the Docker container, the log files are not saved. Data in a Docker container is not ‘persistent’, so you need to make sure you have saved the log data you need onto your host computer before stopping the container.

6.4. Using the Run Controls

Before running Vitaq you need to make sure you have the appropriate clients running as described in the Getting started with Vitaq AI section. For Test Activity diagrams with Scriptworks, you need to start the scriptworks-vitaq-client in the container, by opening a terminal window and typing ‘./scriptwokrs-vitaq-client.js’.

There are four run control buttons provided in the Vitaq tool bar, Run, Pause, Step and Stop, you will see that Run is highlighted and the other three are greyed out. Click on Run. Vitaq will randomly select next allowable Test Actions (based on your Test Activity connections). You will notice the yellow spotlight following the flow from Test Action to Test Action as the test activity runs. This can be seen in both the Test Action Scripts and the Test Activity diagram. The flow arrows are also highlighted to signify that a test activity is running. The Test Activity Output tab displays the Test Actions that have been selected and executed. When a test is running. The Run button becomes greyed out and the other three controls become active.

If you click the Pause button during the run you will see that the pause button is now greyed out and the other three buttons are all enabled. You will also notice that the Test Action that is currently being executed is highlighted by the yellow spotlight in both the diagram and Test Action Scripts windows. The Test Activity Output window will state “Paused Test Activity”, by scrolling back through this window you can see the order of the Test Actions that have been executed.

Now you can click the Step button to step the Test Activity on to select the next allowable Test Action. Step will step you through the Test Action selections one action at a time and is very useful to debug the flow of Test Actions. The Test Activity Output window will state “Test Activity Stepped”. The yellow spotlight will follow the Test Actions as they are been executed. The flow arrows will also be highlighted in yellow, when the step has finished the flow arrows will return back to black, to signify the step has finished executing.

6.5. Click Run and let the Test Activity complete

Each time you click ‘run’ a new seed is created (because we have selected auto-increment for a seed value in the Start Action Properties), and a new randomly selected user journey ‘path’ is generated through the test activity. By randomly selecting which next valid test action to take, the Vitaq Test Activity will auto-generate many valid test action sequences into the software under test. In even reasonably simple software systems there are tens if not hundreds of valid ‘activity’ flows, it is extremely unlikely that the test developers will think of them all and even if they can it is inefficient to write and maintain all the necessary tests. By automating the process, Vitaq will reduce testing time and find bugs that would have otherwise been missed.

The Stop control button terminates the execution and returns the Test Activity to the start position. As the test activity is back at the start state the only run control button that is enabled is Run. The yellow spotlight is no longer valid so is no longer displayed.

6.6. Running a completed Test Activity

We are now ready to complete the bill splitter test activity diagram and run it. It is possible in Vitaq to just test the flow of test activity without defining any test data yet. Complete your connections from test actions to next allowable test actions and enable all the Test Actions except for Settings and Delete, as shown below. This now represents the ‘allowable’ ‘user-journeys’ for the bill splitting web app we are going to test.

TIP To ensure that the flow arrows are connected to flow nodes on a test action in the test activity, select the test action and move it around, all the flow arrows should move as well. If a flow arrow Is not connected then select the arrow by hovering over it, it will turn light blue and show the direction of flow, left click on the blue circle at the arrow head and drag to the flow node on the test activity.

Click on the Test Activity Output tab so that you can see the Test Activity output as it runs, then click on the run button. The Test Activity Output will look like this:

If for any reason the Test Activity does not run, an error code will be displayed in the Script Output tab as shown below.

This Error is telling you to look at the Script Output, so you now need to click on the Script Output tab next to the Test Activity Log tab in the Vitaq window.

This is actually quite a common error. When running Vitaq AI with test Action Scripts using JavaScript Code or Scriptworks Visual coding you need to start the necessary client in the command window of the Docker Container. This error is telling us we have not yet started our javascript client.

The sequence of tracing errors will take you from the Test Activity log to the Script output tab (or script output log file for all of your runs saved to disk) to the client window that you will need for JS code.

6.7. Weighting selection of next allowable Test Actions

When operating Vitaq in ‘exploratory testing’ mode (i.e. without use AI selected in the preferences form) Vitaq’s machine algorithms will direct the selection of next allowable Test Action by the use of weightings on the Test Action links as shown below.

When using Vitaq’s AI mode, this ‘weighting’ is calculated and set automatically by the machine learning algorith to direct the test action selection to meet the user defined goals of ‘user-journey’ sequences, actions and data coverage targets.

For example, to direct the selection of adding a new participant to be a 90% probability when you run the Test Activity, you set the weight of the return arrow of Add_Particpant to 9 and the the weight of arrow going on to expenses to be 1. The weighted probability of selection is then 9/(9+1) for another Add_Participant Test Action selection and 1/(9+1) for the Expenses Test Action to be selected.

6.8. Exhaustive Test Actions

Exhaustive Test Actions are a very powerful feature of Vitaq, they enable users to create a test activity that ensures that next allowable Test Actions connected ‘down stream’ from the exhaustive test action continue to be called ‘exhaustively’ until their call limits are hit.

To show the power of exhaustive test actions we are going to modify the Test Activity with a new test action to display the results of the SplitMe web App expense calculations ‘exhaustively’ until the call limits of the next allowable test actions are reached. Add this test action and call it SplitMe_results and select the Exhaustive checkbox.

Then add two new ‘next-allowable’ test actions, displayExpenses and displayDebts. Then change the name of the Balance action to displayBalance. You can now connect these three ‘next allowable’ test actions to the Splitme_results ‘exhaustive’ test action. This test activity section will exhaustively display the bill split web app results for expenses, balances and debts (exhaustively calling these ‘children test Actions’ until the call limit for each of these actions has been hit. You will notice the Splitme_results test action shows the ‘infinity’ symbol in the bottom right hand corner to denote it is an ‘exhaustive’ test action. The call limit number of this ‘exhaustive’ Test Action is still valid as limit for the number of times this Test Action can be called before being disabled.

Once the Call Limits are met, The Test Activity will stop running as all Test Action calls have been exhausted. You can now ‘run’ the test activity to see what Test Action sequences you get for a number of different new seeds.

6.9. Max Depth

Customised Max Depth settings on Test Actions are only needed when you have not defined a Call Limit on Test Actions. When you define a test action to be exhaustive, it will ‘exhaustively call’ Test Actions down the connected Test Activity until you tell it to stop (i.e. when it hits the Call Limit or reaches a Max Depth limit). When you connect (‘call’) next allowable test actions in a Vitaq test activity recursively (as shown below) you may cause a ‘stack overflow’ issue. Vitaq places the sequence of previously selected test actions on a ‘stack’ when the test activity is being ‘run’. This ‘stack’ can get very deep. In extreme situations this may cause the stack to overflow. This may be resolved by increasing the Max Depth size. The default maximum depth is 10,000, which should suffice for most if not all test activity diagrams. The most common test activity that will cause this sort of overflow is a circular test action selection, as show in the example below where Add_an_Expense and Delete will continually (exhaustively) call each other until the test activity is exhausted. When the Max Depth limit is hit for that test action a Vitaq information message will be issued in the Python Output log and that test action with the max depth limit will no longer be ‘active’ (selectable) in the test activity run.