Falsification of Learning-Based Controllers through Multi-Fidelity Bayesian Optimization

TL;DR

This paper introduces a multi-fidelity Bayesian optimization framework for simulation-based falsification of learning-based controllers, efficiently balancing low- and high-fidelity simulations to identify safety violations cost-effectively.

Contribution

It proposes a novel multi-fidelity Bayesian optimization approach for falsification, optimizing the use of simulators with different fidelity levels to improve efficiency.

Findings

Achieves falsification performance comparable to single-fidelity methods

Reduces computational cost significantly

Automatically determines the optimal fidelity level for evaluation

Abstract

Simulation-based falsification is a practical testing method to increase confidence that the system will meet safety requirements. Because full-fidelity simulations can be computationally demanding, we investigate the use of simulators with different levels of fidelity. As a first step, we express the overall safety specification in terms of environmental parameters and structure this safety specification as an optimization problem. We propose a multi-fidelity falsification framework using Bayesian optimization, which is able to determine at which level of fidelity we should conduct a safety evaluation in addition to finding possible instances from the environment that cause the system to fail. This method allows us to automatically switch between inexpensive, inaccurate information from a low-fidelity simulator and expensive, accurate information from a high-fidelity simulator in a cost-effective way. Our experiments on various environments in simulation demonstrate that multi-fidelity Bayesian optimization has falsification performance comparable to single-fidelity Bayesian optimization but with much lower cost.