8. Functional Coverage

8.1. Introduction

Having learned how to build your Vitaq test activities and run them to auto-generate lots of valuable tests, you may now be deploying Vitaq on as many machines as you can gets your hands on. But how do you know what you are testing?

That is where Functional Coverage can help, but before we launch into the details let’s explain the difference between code coverage and functional coverage. Code coverage checks what lines of codes or expressions and branches in the code have been tested. Functional coverage verifies the functionality of the code based on its functional requirements specification. Combining code coverage and functional coverage is highly recommended to provide an overall metric for testing your system.

Functional coverage is code that observes execution of your test plan. Functional coverage code (also known as a model) tracks whether important values, sets of values, actions, sequences of values or sequences of user journeys have been exercised. It is a very important metric to determine whether you have done enough testing to release your code.

Vitaq can automatically create Functional Coverage models for Test Actions, Test Activity Variables and Test Sequences (user-journeys or use-cases).

These automatically created coverage models are then used as the target goals to drive the Vitaq AI Test Automation. When ‘use AI’ is selected in The Edit->Preferences, Vitaq will use the coverage model as the user-defined goal to achieve 100% coverage with an optimal set of tests.

Vitaq Functional Coverage library methods further enables users to develop functional coverage models capable of automatically monitoring the system being tested. During execution these models determine whether specific use case scenarios have been seen during verification.

As each use case scenario is seen it is logged in an SQL database controlled by Vitaq, providing effective and automated analysis of progress against your test plan.

The Vitaq functional coverage library enables users to map test plan scenarios to functional coverage structures, such as:

• properties of features

• combinations of properties across features

• temporal transitions on properties

• complex temporal scenarios

• Store the collected functional coverage in a postgresql database

• Analyse the database using the Vitaq coverage viewer

8.2. Enabling automatic functional coverage in Vitaq

Vitaq will automatically create a functional coverage model to auto-collect coverage metrics into a postgresql database for Test Actions, Test Activity Variables and Test sequences (for user-journeys or use-cases) by selecting the ‘Use Coverage’ check box in the Edit -> preferences form. To see all the fields you will need to grow the form to its full size by selecting a corner with your mouse and dragging it out.

Vitaq uses postgreSQL, a popular open-source and very fast database. You can define the database admin user name and password in the preferences form. The database name is the name on your Test Activity Diagram tab but converted to lowercase for postgres compatibility. In the example above, we have left the password field blank and hence no password need be entered when viewing the coverage results.

The preference form also gives you the opportunity to set the postgreSQL host ip address and port number. The example above uses the local host and a default port number on the local machine.

8.3. Test Action Coverage

To define your test action goals, click on the Action you want to include in Coverage and select the Action Properties tab on the right. Enter the Action Coverage target number in the form in the tab. For example if you want Vitaq to select the Add_an_Expense action 10 times or more during a Test Activity run then set it to 10 as shown below.

You will notice the coverage symbol appears on the test action with the target value you have set.

Note

If you do not have a coverage target for a particular action (you don’t care how many times that particular action is visited during Vitaq test activity runs, then set the value to 0. No target is then displayed and hence it will not influence the AI driven test automation. A ‘0’ target action will still be executed during test activity runs. If you do not want the action to be executed (because you have not developed the test action script for it yet or that part of your app or software under test is not yet ready, then disable it by clicking in the Enabled box to remove it.) It will then have a ‘no entry sign’.

8.4. Sequence Coverage (User-journey or use case)

To define user-journey (or use case sequence) coverage select the sequence tab on the left next to the test Action Scripts Tab.

Click on the + to start creating your user-journey (or use case sequence) coverage. First enter the name you want to refer to your sequence coverage with (this will be the name it will be recorded in the coverage database as). Below we have user user_journey_1. You then need to click on each action in turn to define the sequence of actions that define your sequence coverage. Notice that incrementing numbers appear on the action selected, denoting the order the actions appears in your sequence coverage definition. If you make a mistake, do not click ok and start again from the +. To change a created sequence coverage you have to delete it and restart from +.

If you have sequences where you do not care which actions are executed in between grouped actions you can define the sequence orders you do care about and leave out the rest. Such as shown below.

Note

If you have not set any targets on your Test Action Coverage when defining sequence coverage that includes these actions, then Vitaq will automatically create a target of 1, so that the sequence overage can be filled.

8.5. Record hit on Error

When Vitaq AI executes Test Action Scripts into the Application Under Test through either Selenium or Appium, there will be circumstances in which an exception is thrown. This might be because of a failing assertion (e.g. checking the value of a field), by an error in the Application Under Test, by an error in the Javascript code or by an explicit “throw” statement. When this happens Vitaq treats this as an error in the test action.

By default Vitaq will not record coverage for a Test Action and hence any Sequence (User Journey) containing the Action that experiences that exception (error).

To change Vitaq AI default behaviour and have it record QA Functional Coverage for Test Actions and hence Sequences that ‘error’, you need to set the ‘Record hit on Error’ in the Preferences file.

Click Edit->Preferences and select the click box as shown below:

8.6. Test Activity Variable Coverage (Data)

The user interface provides the ability of auto-generating a Test Activity Variable Coverage model. For each data type (lists, integers etc) of Test Activity Variables, Vitaq helps you to construct your Data Coverage Targets.

To define targets Functional Coverage has the concept of ‘bins’. A Bin is just a counter that counts the number of times that it has been hit i.e. How many times Vitaq has auto-generated that data value (or values in a range of values.)

Double click on the activity Variable name that you want to define targets for, Vitaq will provide a form for you to enter your Coverage Targets. If your Test Activity Variable is a list of text, then the form will split the list into individual bins for each item of the list as shown below for our new_account_title data.

Warning

If you do not set a target for your Activity Variable bins but ask Vitaq to ‘record’ the data coverage of that Activity Variable you will get a Vitaq error in the Srcipt output such as:

Attribute Error 'Monitor' object has no attribute 'loginname_item'

Where you have created a record Coverage for the loginname Activity Variable but the bins are left at 0, so no targets are set.

For Numbers (Integers or Reals), you have to add your ‘bin’ structure by defining what range of values you want to represent in each bin. In the SplitMe Test Activity we are interested in auto-generating very low expense_values and very high ones (and we are not so interested in all of the values in the middle), so we create bins and bin targets to represent that and give them the targets we want to hit.

8.7. recordCoverage

To make sure that you monitor and measure your data functional coverage for Test Activity Variables into the coverage database, you need to use the recordCoverage method in the Test Action script where you are asking VitaqAI to automatically generate the data using the requestData method.

For JavaScript you need to use the recordCoverage method like this in your script:

await recordCoverage("",`new_account_title`);

Here we are recording Functional Coverage for the Vitaq Test Activity Variable new_account_title. The “”, in the round brackets is just a formatting requirement and hence nothing needs to be inserted between the “”).

Warning

If you fail to record Coverage by not including the method in your script with the appropriate Test Activity Variable, you will not get any Coverage measured for that variable and hence the Vitaq AI algorithms will not be able to achieve 100% Coverage.

Warning

If you record Coverage for a Test Activity Variable before generating data (using the requestData method) then Vitaq AI will error but continue informing that you have not yet generated any Data to measure coverage on.

8.8. Using the Vitaq Coverage viewer

To view and analyse your functional coverage results Vitaq provides a integrated coverage viewer. To open the viewer click on the coverage icon at the top of the Vitaq window.

Select Connect button to ‘connect’ to your Test Activity Coverage database.

The name of the automatically generated Vitaq functional coverage database will be the same as your Test Activity tab name, except all lower case. For example in the Test Activity for the Bill Splitting example we call the Test Activity tab as Bill_Split_Test_Activity in the Vitaq viewer the database will be called bill_split_test_activity as shown above.

The database host is as defined in the preferences i.e. localhost

The user name can either be you user name or the admin name defined at installation. A database password is optional.

Once you have connected to the database you should see the results in the viewer. The Total Coverage view is the accumulated coverage results for all test runs contained in the coverage database. To analyse individual test runs, you can select the individual tests tab and see the list of all test runs. If you wish to delete certain test runs or all test runs from the database then select the individual test run check box on the left side of the individual test run line and then click on the delete selected red dustbin at the top of the viewer. To analyse the individual coverage results in detail click on the right arrow ‘>’ to open out the coverage results for the individual tests.

8.9. Importing an external functional coverage model

To use more advanced functional coverage models that have been created in Python using the extensive list of Vitaq coverage library methods, you need to select the ‘Use imported model’ check box in the edit -> preferences form. This will disable the automatic generation of a functional coverage model by Vitaq and allow you to import the the Python code that describes your new functional coverage model, which is explained in the next section.

8.10. Creating an external functional coverage model

The basic elements of a Vitaq functional coverage model are the Vitaq monitor, the PostgreSQL database, cover groups, items and bins.

The Vitaq monitor is the interface to the database and defines the project parameters. The database stores the test results which can be analysed by the Vitaq viewer so you do not need to write SQL queries.

The Covergroups map Design features, the items map properties of the features and the bins map values of the properties into ‘buckets’. Examples of Covergroups, items and bins can be found in the python code below.

To extend your functional coverage collection to use more advanced capabilities such as cross-coverage and sequence-coverage, you will need to create the Python code using Vitaq functional coverage methods. The easiest way to do this is to first export the python code from your Test Activity and edit this file.

Edit -> Write Python script

Start by cleaning up this exported code to make it into a stand-alone Vitaq functional coverage python model.

Delete the sections for ‘Built-in imports’, ‘redefine sys.stdout and sys.stderr’, Setting up PYTHONPATH and Import Statements. Edit the remaining Vertizan (and 3rd party) imports to leave just these three lines.

from vitaq.core import *

from vitaq.gen import *

from vitaq.cov import *

Then delete all of the code from the Class for the first Test Action genSequence to the end of the file. You should now just have a python coverage model that looks like this:

#!/usr/bin/env python
# ------------------------------------------------------------------------------
#  Generated using main.js at Thu, 23 May 2019 11:09:23 GMT
# ------------------------------------------------------------------------------
"""
Bill_Split_Test_Activity.py: Vitaq Python test code generated from Bill_Split_Test_Activity GUI representation
"""

# Vertizan (and 3rd party) imports
# ------------------------------------------------------------------------------
from vitaq.core import *
from vitaq.gen import *
from vitaq.cov import *

# Class: CoverageModel
# ==============================================================================
class CoverageModel(Monitor):
    '''
    CoverageModel: Coverage model for the Bill_Split_Test_Activity Vitaq test
    '''

    # --------------------------------------------------------------------------
    def __init__(self):
        '''
        __init__: Coverage model initialisation
        '''

        # Initialise the monitor - do not change this format if using this for an external model
        # substitution with correct values relies on this format.
        # --------------------------------------------------------------------------
        db_name = 'bill_split_test_activity'
        db_type = str2sql_database_type(get_str_env("VERTIZAN_DB_KIND"))
        db_server = get_str_env("VERTIZAN_DB_HOST")
        db_user = "sean"
        db_passwd = ''
        test_name = 'Bill_Split_Test_Activity_test'
        project_name = 'Bill_Split_Test_Activity_proj'
        super(CoverageModel, self).__init__(db_type, db_name, db_server, db_user, db_passwd, test_name, project_name)

        # Define the float variables
        # --------------------------------------------------------------------------
        self.float_variables = {
            'Expense_value':      2,
        }

        # Create the structure to record the cross_items
        # --------------------------------------------------------------------------
        # This is a dict of lists
        self.cross_items = {}
        

        # Create the covergroups
        # --------------------------------------------------------------------------
        self.variables_cg = Covergroup(self, 'Variables', 1)
        self.actions_cg = Covergroup(self, 'Actions', 1)

        # Add_Participant_Name
        # --------------------------------------------------------------------------
        self.add_participant_name = VitaqVar(dtype='int64')
        self.add_participant_name_item = self.variables_cg.add_item('''Add_Participant_Name''', self.add_participant_name, dtype='int64')
        self.add_participant_name_item.add_bin('''John''', 0, 1, 10)
        self.add_participant_name_item.add_bin('''Susan''', 1, 1, 10)
        self.add_participant_name_item.add_bin('''Roger''', 2, 1, 10)
        self.add_participant_name_item.add_bin('''Ross''', 3, 1, 10)
        self.add_participant_name_item.add_bin('''Charles''', 4, 1, 10)
        self.add_participant_name_item.add_bin('''Sean''', 5, 1, 10)
        self.add_participant_name_item.add_bin('''James''', 6, 1, 10)
        self.add_participant_name_item.add_bin('''Nambi''', 7, 1, 10)

        # Expense_item
        # --------------------------------------------------------------------------
        self.expense_item = VitaqVar(dtype='int64')
        self.expense_item_item = self.variables_cg.add_item('''Expense_item''', self.expense_item, dtype='int64')
        self.expense_item_item.add_bin('''Breakfast''', 0, 1, 10)
        self.expense_item_item.add_bin('''Lunch''', 1, 1, 10)
        self.expense_item_item.add_bin('''Dinner''', 2, 1, 10)
        self.expense_item_item.add_bin('''Snacks''', 3, 1, 10)
        self.expense_item_item.add_bin('''Drinks''', 4, 1, 10)
        self.expense_item_item.add_bin('''Cinema_tickets''', 5, 1, 10)
        self.expense_item_item.add_bin('''Train_tickets''', 6, 1, 10)
        self.expense_item_item.add_bin('''Theatre_tickets''', 7, 1, 10)
        self.expense_item_item.add_bin('''Bus_tickets''', 8, 1, 10)

        # Expense_value
        # --------------------------------------------------------------------------
        self.expense_value = VitaqVar(dtype='int64')
        self.expense_value_item = self.variables_cg.add_item('''Expense_value''', self.expense_value, dtype='int64')
        self.expense_value_item.add_bin('''0.00-20.00''', 0, 2000, 1, 10)
        self.expense_value_item.add_bin('''20.01-40.00''', 2001, 4000, 1, 10)
        self.expense_value_item.add_bin('''40.01-60.00''', 4001, 6000, 1, 10)
        self.expense_value_item.add_bin('''60.01-80.00''', 6001, 8000, 1, 10)
        self.expense_value_item.add_bin('''80.01-100.00''', 8001, 10000, 1, 10)

        # New_Account_Description
        # --------------------------------------------------------------------------
        self.new_account_description = VitaqVar(dtype='int64')
        self.new_account_description_item = self.variables_cg.add_item('''New_Account_Description''', self.new_account_description, dtype='int64')
        self.new_account_description_item.add_bin('''A description''', 0, 1, 10)
        self.new_account_description_item.add_bin('''B description''', 1, 1, 10)
        self.new_account_description_item.add_bin('''Aaaaaa veeeeeerrrryyyyy Loooooooooonggggggggggg description''', 2, 1, 10)

        # New_Account_Title
        # --------------------------------------------------------------------------
        self.new_account_title = VitaqVar(dtype='int64')
        self.new_account_title_item = self.variables_cg.add_item('''New_Account_Title''', self.new_account_title, dtype='int64')
        self.new_account_title_item.add_bin('''Friends''', 0, 1, 10)
        self.new_account_title_item.add_bin('''Family''', 1, 1, 10)
        self.new_account_title_item.add_bin('''Work''', 2, 1, 10)

        # New_Account_Your_Name
        # --------------------------------------------------------------------------
        self.new_account_your_name = VitaqVar(dtype='int64')
        self.new_account_your_name_item = self.variables_cg.add_item('''New_Account_Your_Name''', self.new_account_your_name, dtype='int64')
        self.new_account_your_name_item.add_bin('''Meital''', 0, 1, 10)
        self.new_account_your_name_item.add_bin('''Simon''', 1, 1, 10)
        self.new_account_your_name_item.add_bin('''Nambi''', 2, 1, 10)
        self.new_account_your_name_item.add_bin('''Charles''', 3, 1, 10)
        self.new_account_your_name_item.add_bin('''Susan''', 4, 1, 10)
        self.new_account_your_name_item.add_bin('''Ross''', 5, 1, 10)
        self.new_account_your_name_item.add_bin('''John''', 6, 1, 10)
        self.new_account_your_name_item.add_bin('''James''', 7, 1, 10)
        self.new_account_your_name_item.add_bin('''Sean''', 8, 1, 10)

        # Add_Participant
        # --------------------------------------------------------------------------
        self.add_participant = VitaqVar(dtype='int64')
        self.add_participant_item = self.actions_cg.add_item('''Add_Participant''', self.add_participant, dtype='int64')
        self.add_participant_item.add_bin('''Add_Participant''', 1, 1, 10)

        # Expenses
        # --------------------------------------------------------------------------
        self.expenses = VitaqVar(dtype='int64')
        self.expenses_item = self.actions_cg.add_item('''Expenses''', self.expenses, dtype='int64')
        self.expenses_item.add_bin('''Expenses''', 1, 1, 10)

        # Add_an_Expense
        # --------------------------------------------------------------------------
        self.add_an_expense = VitaqVar(dtype='int64')
        self.add_an_expense_item = self.actions_cg.add_item('''Add_an_Expense''', self.add_an_expense, dtype='int64')
        self.add_an_expense_item.add_bin('''Add_an_Expense''', 1, 1, 10)

        # New_Account
        # --------------------------------------------------------------------------
        self.new_account = VitaqVar(dtype='int64')
        self.new_account_item = self.actions_cg.add_item('''New_Account''', self.new_account, dtype='int64')
        self.new_account_item.add_bin('''New_Account''', 1, 1, 10)

        # Balance
        # --------------------------------------------------------------------------
        self.balance = VitaqVar(dtype='int64')
        self.balance_item = self.actions_cg.add_item('''Balance''', self.balance, dtype='int64')
        self.balance_item.add_bin('''Balance''', 1, 1, 10)

        # Save
        # --------------------------------------------------------------------------
        self.save = VitaqVar(dtype='int64')
        self.save_item = self.actions_cg.add_item('''Save''', self.save, dtype='int64')
        self.save_item.add_bin('''Save''', 1, 1, 10)

        # Quit
        # --------------------------------------------------------------------------
        self.quit = VitaqVar(dtype='int64')
        self.quit_item = self.actions_cg.add_item('''Quit''', self.quit, dtype='int64')
        self.quit_item.add_bin('''Quit''', 1, 1, 10)

        # Cancel
        # --------------------------------------------------------------------------
        self.cancel = VitaqVar(dtype='int64')
        self.cancel_item = self.actions_cg.add_item('''Cancel''', self.cancel, dtype='int64')
        self.cancel_item.add_bin('''Cancel''', 1, 1, 10)

        # Construct everything in the db we just defined
        # --------------------------------------------------------------------------
        self.construct_all_coverage_sql()
        self.update_count = 0
        

    # --------------------------------------------------------------------------
    def __del__(self):
        '''
        __del__: Wrap-up the coverage
        '''
        del self
    

    # --------------------------------------------------------------------------
    def record(self, data):
        '''
        record: Coverage recording
        '''

        self.update_count += 1
        for variable, value in iteritems(data):
            if variable in self.float_variables.keys():
                value = value * 10**self.float_variables[variable]
            variable = variable.lower()
            if hasattr(self, '{}'.format(variable)):
                eval('self.{}.set_value({})'.format(variable, int(value)))
                eval('self.{}_item.hit()'.format(variable))
        
        # Hit the cross items
        for (cross_item, required_variables) in iteritems(self.cross_items):
            if all(variable in data for variable in required_variables):
                cross_item.hit()
            
        if self.update_count >= 100:
            self.update_count = 0
            self.update_all_modified_bins_sql()