Surprise Potential as a Measure of Interactivity in Driving Scenarios

TL;DR

This paper introduces a novel surprise potential metric to identify interactive driving scenarios in autonomous vehicle logs, improving the selection of critical situations for safety validation and outperforming existing methods.

Contribution

The paper proposes a new surprise potential measure based on three design dimensions, evaluated exhaustively on nuScenes, and validated with human-aligned reward correlation.

Findings

Surprise potential correlates with human preferences at over 0.82.

The measure outperforms existing approaches in identifying interactive scenarios.

Validated motion planners on curated scenarios demonstrate practical utility.

Abstract

Validating the safety and performance of an autonomous vehicle (AV) requires benchmarking on real-world driving logs. However, typical driving logs contain mostly uneventful scenarios with minimal interactions between road users. Identifying interactive scenarios in real-world driving logs enables the curation of datasets that amplify critical signals and provide a more accurate assessment of an AV's performance. In this paper, we present a novel metric that identifies interactive scenarios by measuring an AV's surprise potential on others. First, we identify three dimensions of the design space to describe a family of surprise potential measures. Second, we exhaustively evaluate and compare different instantiations of the surprise potential measure within this design space on the nuScenes dataset. To determine how well a surprise potential measure correctly identifies an interactive scenario, we use a reward model learned from human preferences to assess alignment with human intuition. Our proposed surprise potential, arising from this exhaustive comparative study, achieves a correlation of more than 0.82 with the human-aligned reward function, outperforming existing approaches. Lastly, we validate motion planners on curated interactive scenarios to demonstrate downstream applications.