Complete Requirements-based Testing with Finite State Machines

TL;DR

This paper introduces new requirements-based testing methods for deterministic finite state machines, providing strategies with proven fault coverage guarantees suitable for safety-critical systems.

Contribution

It presents two novel test generation strategies for requirements-based testing with formal guarantees, and discusses their practical applicability and complexity considerations.

Findings

First strategy guarantees requirement satisfaction if passed

Second strategy guarantees requirement satisfaction if and only if passed

Preferred for safety-critical systems due to thoroughness

Abstract

In this paper, new contributions to requirements-based testing with deterministic finite state machines are presented. Elementary requirements are specified as triples consisting of a state in the reference model, an input, and the expected reaction of the system under test defined by a set of admissible outputs, allowing for different implementation variants. Composite requirements are specified as collections of elementary ones. Two requirements-driven test generation strategies are introduced, and their fault coverage guarantees are proven. The first is exhaustive in the sense that it produces test suites guaranteeing requirements satisfaction if the test suite is passed. If the test suite execution fails for a given implementation, however, this does not imply that the requirement has been violated. Instead, the failure may indicate an arbitrary violation of I/O-equivalence, which could be unrelated to the requirement under test. The second strategy is complete in the sense that it produces test suites guaranteeing requirements satisfaction if and only if the suite is passed. Complexity considerations indicate that for practical application, the first strategy should be preferred to the second. Typical application scenarios for this approach are safety-critical systems, where safety requirements should be tested with maximal thoroughness, while user requirements might be checked with lesser effort, using conventional testing heuristics.