16. Vitaq Python scripting Methodology

16.1. Introduction

A Vitaq Test Activity should always act as an informative bug report when it fails, so we recommend that you keep in mind these five questions when using Vitaq.

• What is the unit or system under test?

• What should your Test Activity do? (Prose description)

• What was the actual output?

• What was the expected output?

• How do you reproduce the failure?

Let’s take each of these questions in turn and explain what we provide in Vitaq to help.

16.2. What is the unit or system under test?

Give your Vitaq_Test_Activity.vtq file a meaningful name. If you have developed a user requirements specification then use a name that links the Test Activity to the features you are testing in the spec.

16.3. What should your Test Activity do?

Use the Activity_start Test Action Script to document what the Test Activity should do using Python comments in the code and/or printing to the stdout which will be seen in the Python Output window.

16.4. What was the actual output?

Vitaq provides two reporting windows Vitaq output windows the Test Activity Output tab and the Python Output tab. The Test Activity Output tab is a useful reporting view on Test Actions and Test Activity variables. All of the Test Activity Script output for stdout and stderr is by default provided directly into the Python Output window. This information can be redirected into external log files using the methods below.

16.5. Redirecting your Test Activity output to a log file

The Vitaq library methods Appendix 1a: Vitaq Library Methods - JavaScript have a number of functions that help with reporting including redirect_stdout redirect_stderr and redirect_stderr redirect_stderr

self.redirect_stdout('../output/Test_GUI_003_25_0_stdout.txt') self.redirect_stderr('../output/Test_GUI_003_25_0_stderr.txt')

16.6. What was the expected output?

Vitaq will auto-generate a large varied set of input tests which is great for robustness testing and can be left running until your software under test throws an error. Unfortunately just waiting for the software to error does not prove that everything worked as expected. As a test runs the resulting values created by your software such as a function returning a value will need to be checked.

Vitaq provides a simple and convenient check library method check and in many cases this will be all you will need.

To achieve a greater level of checking we use assertions. We access assertions by using the python unittest assertion library. It provides a very comprehensive set of assertions.

16.7. Checking using Assertions

The Python unittest library https://docs.python.org/2/library/unittest.html provides a rich set of assertions that Vitaq provide access to.

To use any of these assertion methods in Vitaq Test Action scripts you must first import the unittest library into Vitaq using the imports Test Action script

import unittest

Then in the Action_Start Test Action Script type

self.assertions = unittest.TestCase('__init__')

When ever an assertion is needed in a Test Action Script it is referenced using self.parent.assertions.

For example:

self.parent.assertions.assertEqual(variable_a,variable_b,msg='variable_a does not equal variable_b')

When using an assertion in Vitaq Test Action Scripts we recommend that you place them in a ‘try and except Block’ which is used in Python to catch and handle exceptions. Python executes the code following the try statement as a ‘normal’ part of the Test Action Script. The code that follows the except statement is the program’s response to any exceptions in the preceding try clause.

So in the code example below:

In the normal flow of the Test Action Script Vitaq will run the assertion to see if the two variables (splitme_app_balance_value and participant_balance) are equal. The code in the try clause will stop as soon as an exception is encountered, if the two variables are not equal then the assertion will raise an exception and run the code following the except clause. The message defined in the assertion can then be printed to the stdout window and the assert message ‘msg’ can be printed using print(err). To make sure Vitaq does not stop on an error, you can use the Vitaq library method ‘error_continue’ error_continue as shown in the example below.

try:

# Test that splitme_app_balance_value and participant_balance are equal. # If the values do not compare equal, the test will fail. self.parent.assertions.assertEqual(splitme_app_balance_value,

participant_balance, msg='***TEST HAS FAILED*** Assertion check fired')

except AssertionError as err:

print('Participant balance calculated by App is incorrect') print(err) self.info('Info: {}'.format(err)) self.error_continue('Participant balance value difference: {}'

.format(splitme_app_balance_value-participant_balance))

return False

16.8. Python Assertions

A list of the most commonly used Python assertions provided in the unittest library are listed below. The complete list can be found in the online docs https://docs.python.org/2/library/unittest.html

Python assertions

Method	Checks
assertEqual(a, b)	a == b
assertNotEqual(a, b)	a != b
assertTrue(x)	bool(x) is True
assertFalse(x)	bool(x) is False
assertIs(a, b)	a is b
assertIsNot(a, b)	a is not b
assertIsNone(x)	x is None
assertIsNotNone(x)	x is not None
assertIn(a, b)	a in b
assertNotIn(a, b)	a not in b
assertIsInstance(a, b)	isinstance(a, b)
assertNotIsInstance(a, b)	notisinstance(a, b)

16.9. Checking using Oracles (golden reference model)

Test oracles are a great way to check that the software under test (SUT) is functioning correctly. By using an oracle it means that the checking of a test run can be automated. As Vitaq can/will generate large numbers of tests it is very important that the results can be checked automatically without human intervention, significantly speeding up the testing process.

A test oracle is the gold standard for the SUT, in that it is considered the correct answer, if the SUT differs from the oracle the SUT is at fault. It might specify correct output for all possible inputs or only for specific inputs. It might not specify actual output values but only constraints on them.

Oracles are created as a program in there own right and are separate from the system under test.

For example in our bill splitting App example, we could use a spreadsheet program to calculate the bill split balance to compare against the result calculated in the web App. This is described in the sections Writing into excel spreadsheets from Test Action Scripts and Reading from excel spreadsheets into Vitaq Test Action Scripts

16.10. How do you reproduce the failure?

Checks will help you determine if you have an error, but Vitaq also provides some useful methods to help you manage errors such as get_error_count() get_error_count These types of ‘tear-down’ methods should be placed into the Activity_end Test Action Script.

For Example:

total_number_of_errors = self.get_error_count() number_of_errors = str(total_number_of_errors) self.info('Total number of errors = {}'.format(number_of_errors))

Each time you run your Test Activity with a different seed value (see Setting Seeds) Vitaq’s machine algorithms will automatically select different next allowable Test Actions and generate different Test Activity Variable values. This results in many different and valid tests being created. When a test fails you need to be able to reproduce exactly the circumstances that led to the failure. This is a key feature of Vitaq, when the same seed value with the same test activity is run exactly the same test result will be achieved. We call this random stability, same seed, same test, same result. Therefore by keeping track of the seed value used and the pass/fail result of a test, debugging of failed tests can be carried out.

16.11. Seed management

Vitaq provides a very simple method for keeping track of the seed value used in a test run. It is the Vitaq get_seed() method get_seed. To use it add the lines below to your Activiy_Start Test Action Script, the seed value will be output to the Python Output window.

For example:

seed_value = self.get_seed()

self.info('seed = {}'.format(seed_value))