Probably Approximately Correct (PAC) Guarantees for Data-Driven Reachability Analysis: A Theoretical and Empirical Comparison

TL;DR

This paper compares data-driven reachability analysis methods that provide probabilistic safety guarantees, highlighting their differences, trade-offs, and practical considerations through theoretical and empirical analysis.

Contribution

It establishes a formal connection between PAC guarantees of various techniques and provides practical guidance on their application.

Findings

Different methods have subtle interpretational differences.

Trade-offs exist between computational effort and sample size.

Methods are not directly interchangeable despite formal similarities.

Abstract

Reachability analysis evaluates system safety, by identifying the set of states a system may evolve within over a finite time horizon. In contrast to model-based reachability analysis, data-driven reachability analysis estimates reachable sets and derives probabilistic guarantees directly from data. Several popular techniques for validating reachable sets -- conformal prediction, scenario optimization, and the holdout method -- admit similar Probably Approximately Correct (PAC) guarantees. We establish a formal connection between these PAC bounds and present an empirical case study on reachable sets to illustrate the computational and sample trade-offs associated with these methods. We argue that despite the formal relationship between these techniques, subtle differences arise in both the interpretation of guarantees and the parameterization. As a result, these methods are not generally interchangeable. We conclude with practical advice on the usage of these methods.