Part-X: A Family of Stochastic Algorithms for Search-Based Test Generation with Probabilistic Guarantees

TL;DR

This paper introduces a family of stochastic algorithms for search-based test generation that provides probabilistic guarantees on finding errors within finite testing budgets, addressing a key limitation of existing falsification methods.

Contribution

It develops a new stochastic algorithm that estimates the probability of falsifying behaviors and identifies potential regions, enabling finite-time guarantees in testing.

Findings

Effective in standard benchmark functions

Successfully applied to F16 benchmark problem

Provides probabilistic bounds on falsification success

Abstract

Requirements driven search-based testing (also known as falsification) has proven to be a practical and effective method for discovering erroneous behaviors in Cyber-Physical Systems. Despite the constant improvements on the performance and applicability of falsification methods, they all share a common characteristic. Namely, they are best-effort methods which do not provide any guarantees on the absence of erroneous behaviors (falsifiers) when the testing budget is exhausted. The absence of finite time guarantees is a major limitation which prevents falsification methods from being utilized in certification procedures. In this paper, we address the finite-time guarantees problem by developing a new stochastic algorithm. Our proposed algorithm not only estimates (bounds) the probability that falsifying behaviors exist, but also it identifies the regions where these falsifying behaviors may occur. We demonstrate the applicability of our approach on standard benchmark functions from the optimization literature and on the F16 benchmark problem.