10. Continuous Integration

10.1. CLI For Vitaq CI

It is a common requirement to be able to run the Vitaq AI Test Activity as part of a continuous integration tool such as Jenkins, BitRise or Circle CI. To do so requires being able to launch the execution of the Test Activity from the CI tool as a scripted command, and then recover results that can be analysed and displayed in the CI tool. Often Vitaq will be installed on a separate machine from the CI server or in the cloud.

At a high level, the Test Activity can be saved as a standalone Python script. This can then be executed with a “curl” call and the results retrieved.

10.2. Creating the “headless” Vitaq Test Script

The first step is to save the Vitaq Test Activity (.vtq file) that you wish to run as part of your Continuous Integration. Be sure that the Test Activity runs through the UI as you would expect it to and that you have set the range of seeds that you wish to execute in the CI (if you are running exploratory testing) or you have selected AI (to achieve maximum QA coverage).

The next step is to save a Python script which we will run. To do this go to the menu bar and choose Edit->Write Python Script. A dialog will ask you for a name and location for saving the file. This script will be identical to the script that is created when running from the Vitaq UI, except it does not have the code that connects it to the UI, so it will run standalone i.e. headless.

10.3. Executing the headless Test Activity from CI when using vitaq_client

To run the headless Test Activity and receive back results in an XML format we use the following curl command from the CI tool outside the container or for testing purposes from a shell.

curl -d '{"testScriptPy": "<path to headless python script>",
          "pythonExecutable": "<path to python executable>",
          "workDir": "<path to the directory where the script should run>",
     -H "Content-Type: application/json"
     -X POST '<ip address of Vitaq machine>:<port number of client>/ci/runTests'
     -o results.xml

Where:

-d

  • Specifies a JSON data structure which contains:

<path to headless python script>

  • is the path to the Python script that we saved in the step above, using the file system of the Vitaq machine

<path to python executable>

  • the path to the Python executable on the Vitaq machine

<path to the directory where the script should run>

  • where the script should run on the Vitaq machine

-H

– Specifies that we are sending JSON content

-X

– Send this to the HTTP server/route at this URL <ip address of Vitaq machine>

  • IP of the Vitaq machine. If using a Docker container, this will be the IP of the Docker host and port mapping will need to be used.

<port number of client>

  • the port number of the scriptworks-vitaq-client (normally 8082). If using a Docker container, this port will need to be mapped to the docker host using ‘-p 8082:8082’ in the docker run command.

-o

– Specify the name for a local output file – this will be an artifact of this step in the CI flow

Note

You would need to give the path to the workDir if it is not the user home directory.

Note

With regard to the IP:port value, the IP address is the IP address of the machine where the container is running and the port is the port that the Vitaq_client is running on (default to 8082). You will need to map the port to the host machine when you start the docker container using “-p 8082:8082”.

Note

The results you get back will be a JSON structure representing the coverage.

An example of this when executing your CI is shown below (note all of the paths are given in Unix Syntax)

curl -d '{"testScriptPy": "/home/vitaq/SauceDemoSelenium.py", "pythonExecutable": "/usr/bin/python3.7", "workDir": "."}' -H "Content-Type: application/json" -X POST '192.168.0.106:8082/ci/runTests' -o results.xml

10.4. Executing the headless Test Activity from CI when using scriptworks_vitaq_client

The first step is to run the headless Test Activity and receive back results in an XML format. To do this we use the following curl command from the CI tool outside the container or for testing purposes from a shell.

curl -d '{"testScriptPy": "<path to headless python script>",
          "pythonExecutable": "<path to python executable>",
          "workDir": "<path to the directory where the script should run>",
                      "outputFormat": "<format of XML result>" }'
     -H "Content-Type: application/json"
     -X POST '<ip address of Vitaq machine>:<port number of client>/ci/runTests'
     -o results.xml

Where:

-d

  • Specifies a JSON data structure which contains:

<path to headless python script>

  • is the path to the Python script that we saved in the step above, using the file system of the Vitaq machine

<path to python executable>

  • the path to the Python executable on the Vitaq machine

<path to the directory where the script should run>

  • where the script should run on the Vitaq machine

<format of XML result>

  • this can be either JUnit (a basic JUnit format XML) or ExtJUnit (Junit format but with more detail).

-H

– Specifies that we are sending JSON content

-X

– Send this to the HTTP server/route at this URL <ip address of Vitaq machine>

  • IP of the Vitaq machine. If using a Docker container, this will be the IP of the Docker host and port mapping will need to be used.

<port number of client>

  • the port number of the scriptworks-vitaq-client (normally 8082). If using a Docker container, this port will need to be mapped to the docker host using ‘-p 8082:8082’ in the docker run command.

-o

– Specify the name for a local output file – this will be an artifact of this step in the CI flow

An example of this when executing your CI on a Windows machine is shown below

curl -d '{"testScriptPy": "/home/vitaq/SauceDemoSelenium.py", "pythonExecutable": "/usr/bin/python3.7", "workDir": "."}' -H "Content-Type: application/json" -X POST '192.168.0.106:8082/ci/runTests' -o results.xml

10.5. Converting the XML result to HTML for display in the CI tool

The next step is to send the XML back to the client for conversion to HTML which provides the results in a format that can be displayed in the CI tool and more easily navigated.

curl -d "@<name of XML file saved in previous step>"
     -H "Content-Type: text/plain"
     -X POST '<ip address of Vitaq machine>:<port number of client>/ci/convertXml'
     -o results.html

Where:

-d

  • Specifies a plain text data to send

<name of XML file saved in previous step>

  • name of the file created as output on the previous step.

-H

– Specifies that we are sending plain text

-X

– Send this to the HTTP server/route at this URL <ip address of Vitaq machine>

  • IP of the Vitaq machine. If using a Docker container, this will be the IP of the Docker host and port mapping will need to be used.

<port number of client>

  • the port number of the scriptworks-vitaq-client (normally 8082). If using a Docker container, this port will need to be mapped to the docker host using ‘-p 8082:8082’ in the docker run command.

-o

– Specify the name for a local output file – this will be an artifact of this step in the CI flow

An example of this on a Windows machine is shown below

curl -d "@results.xml" -H "Content-Type: text/plain" -X POST '192.168.0.106:8082/ci/convertXml' -o results.html