Querying Labelled Data with Scenario Programs for Sim-to-Real Validation

TL;DR

This paper introduces a formal method using scenario programs to validate whether failure scenarios identified in simulation of autonomous vehicles are reproducible in real-world data, addressing the sim-to-real gap.

Contribution

It proposes a formal definition and a querying algorithm to match labelled real-world data with abstract scenario programs for sim-to-real validation.

Findings

Algorithm is accurate and efficient on realistic traffic scenarios

Scales well with a reasonable number of agents

Provides a formal framework for scenario validation

Abstract

Simulation-based testing of autonomous vehicles (AVs) has become an essential complement to road testing to ensure safety. Consequently, substantial research has focused on searching for failure scenarios in simulation. However, a fundamental question remains: are AV failure scenarios identified in simulation meaningful in reality, i.e., are they reproducible on the real system? Due to the sim-to-real gap arising from discrepancies between simulated and real sensor data, a failure scenario identified in simulation can be either a spurious artifact of the synthetic sensor data or an actual failure that persists with real sensor data. An approach to validate simulated failure scenarios is to identify instances of the scenario in a corpus of real data, and check if the failure persists on the real data. To this end, we propose a formal definition of what it means for a labelled data item to match an abstract scenario, encoded as a scenario program using the SCENIC probabilistic programming language. Using this definition, we develop a querying algorithm which, given a scenario program and a labelled dataset, finds the subset of data matching the scenario. Experiments demonstrate that our algorithm is accurate and efficient on a variety of realistic traffic scenarios, and scales to a reasonable number of agents.