Efficient Determination of Safety Requirements for Perception Systems

TL;DR

This paper introduces a new method called smoothing bandits to efficiently determine safe perception system performance requirements within autonomous systems, demonstrated on aircraft collision avoidance with improved accuracy and efficiency.

Contribution

The paper presents a novel estimation technique combining Gaussian processes and threshold bandits for safety requirement determination in perception systems.

Findings

Improved accuracy over baseline methods

Enhanced efficiency in safety estimation

Effective application to aircraft collision avoidance

Abstract

Perception systems operate as a subcomponent of the general autonomy stack, and perception system designers often need to optimize performance characteristics while maintaining safety with respect to the overall closed-loop system. For this reason, it is useful to distill high-level safety requirements into component-level requirements on the perception system. In this work, we focus on efficiently determining sets of safe perception system performance characteristics given a black-box simulator of the fully-integrated, closed-loop system. We combine the advantages of common black-box estimation techniques such as Gaussian processes and threshold bandits to develop a new estimation method, which we call smoothing bandits. We demonstrate our method on a vision-based aircraft collision avoidance problem and show improvements in terms of both accuracy and efficiency over the Gaussian process and threshold bandit baselines.