Synthesizing Adaptive Test Strategies from Temporal Logic Specifications

TL;DR

This paper introduces a formal method-based approach to generate adaptive test strategies from temporal logic specifications, capable of revealing faults in reactive systems under various fault assumptions, demonstrated on satellite system components.

Contribution

It presents a novel synthesis method for adaptive test strategies from temporal logic specifications using reactive synthesis with partial information, applicable to industrial systems.

Findings

Strategies can detect hard-to-find bugs in real-world systems

Approach supports different fault assumptions, from transient to persistent

Effective on industrial safety-critical components

Abstract

Constructing good test cases is difficult and time-consuming, especially if the system under test is still under development and its exact behavior is not yet fixed. We propose a new approach to compute test strategies for reactive systems from a given temporal logic specification using formal methods. The computed strategies are guaranteed to reveal certain simple faults in every realization of the specification and for every behavior of the uncontrollable part of the system's environment. The proposed approach supports different assumptions on occurrences of faults (ranging from a single transient fault to a persistent fault) and by default aims at unveiling the weakest one. Based on well-established hypotheses from fault-based testing, we argue that such tests are also sensitive for more complex bugs. Since the specification may not define the system behavior completely, we use reactive synthesis algorithms with partial information. The computed strategies are adaptive test strategies that react to behavior at runtime. We work out the underlying theory of adaptive test strategy synthesis and present experiments for a safety-critical component of a real-world satellite system. We demonstrate that our approach can be applied to industrial specifications and that the synthesized test strategies are capable of detecting bugs that are hard to detect with random testing.