Automated (UI) testing with Matrix Requirements

Preamble

From an auditors point of view a verification test is something done by a human. A verification test needs criteria for acceptance, and a final evaluation, whether a test was passed or not, and what to do in case deviations occured. That is why all test reports need a stated test result, and a responsible person for the final result decision.
Therefore you cannot just rely on some automated test reports created by a devops tool or continous integration server.

An executed test needs to have information what was tested, who did it, checks that all steps have been executed (with pass fail information) and some objective evidence (that can be screenshots, generated documents, links to found issues, or other info). Each test run needs a final statement about the overall test result, and justification for the statement, if that is not obvious, according to the acceptance criteria.

Modern Testing

Today testing is often already done during software development using automated tests running after committing a code change. While these continuously executed tests are super important for the quality of the product they are not useful for the verification report. In the verification and validation plan you need set the acceptance criteria, for instance a goal on how much of the code is covered by these tests ("code coverage"), and how you make sure this is measurable. Also, you need to describe how you evaluate code coverage of tests through all code updates and branches. For the release documentation it is important to state how much of the code was actually covered and that the tests passed for the release. The "manual" verification can be done with a specific code quality test hat has two test steps: verify the realized coverage by a specified method (> x% covered), evaluate the automatic test results (no / only acceptable failures). Both need to be evaluated according to the previous acceptance criteria (goal for the code coverage incl. the measurement method, and acceptable deviations).

Combining the auditors expectations and modern testing

The idea is to execute the tests manually with support of a testing frame work. In this example we use cypress.io 

Imagine you have a test case with the following test steps:

Let's now assume you can automate the bold steps above. With Matrix Requirements you could add some scripts that do the test action and fill the evidence:

After the first test steps have been executed the tester fills the pass/fail of automated steps and then  manually executes steps 2 and 3:

And then lets the rest of the scripts execute:Once done the tester can evaluate the last two automated results:

According to the results of each test step, the overall test result of the test will be set in the Matrix Test report item (XTC). Here, the tester can decide, whether a test is overall passed, even if a problem occured in any of the test steps. 

That way, the verification test will be documented by a tester, although testing has been done by scripts.

According to the example, larger test scripts can be used for automated tests of complete SW branches. In that case, the test step will be the complete run of an automated branch test. The manual setting of the test result as pass/fail for the test step is the evaluation of the results of the automated tests according to the acceptance criteria.

That is what auditors want to see. For each automated branch test, the criteria "code coverage" and "only acceptable deviations" should be the pass/fail criteria.