A Formal Characterization of Black-Box System Safety Performance with Scenario Sampling

TL;DR

This paper introduces a formal framework for evaluating black-box system safety using scenario sampling, proposing an optimal algorithm to estimate safe operational domains with high accuracy and demonstrating its effectiveness through a robot collision avoidance case study.

Contribution

It presents a novel safety evaluation criterion and an asymptotically optimal scenario sampling algorithm for black-box systems, with theoretical guarantees and practical validation.

Findings

The proposed algorithm converges to the true safe operational domain.

It remains effective under biased sampling strategies.

The approach is validated on a mobile robot collision avoidance system.

Abstract

A typical scenario-based evaluation framework seeks to characterize a black-box system's safety performance (e.g., failure rate) through repeatedly sampling initialization configurations (scenario sampling) and executing a certain test policy for scenario propagation (scenario testing) with the black-box system involved as the test subject. In this letter, we first present a novel safety evaluation criterion that seeks to characterize the actual operational domain within which the test subject would remain safe indefinitely with high probability. By formulating the black-box testing scenario as a dynamic system, we show that the presented problem is equivalent to finding a certain "almost" robustly forward invariant set for the given system. Second, for an arbitrary scenario testing strategy, we propose a scenario sampling algorithm that is provably asymptotically optimal in obtaining the safe invariant set with arbitrarily high accuracy. Moreover, as one considers different testing strategies (e.g., biased sampling of safety-critical cases), we show that the proposed algorithm still converges to the unbiased approximation of the safety characterization outcome if the scenario testing satisfies a certain condition. Finally, the effectiveness of the presented scenario sampling algorithms and various theoretical properties are demonstrated in a case study of the safety evaluation of a control barrier function-based mobile robot collision avoidance system.