Plane and Sample: Maximizing Information about Autonomous Vehicle Performance using Submodular Optimization

TL;DR

This paper introduces a submodular optimization framework for scenario sampling in autonomous vehicle evaluation, improving efficiency and transferability across operational domains by leveraging information gain and Bayesian modeling.

Contribution

It reformulates scenario sampling as a submodular optimization problem using Bayesian Hierarchical Models, enabling efficient and transferable AV performance evaluation.

Findings

Only 7.5% of scenarios needed for evaluation, a 23% improvement over Latin Hypercube Sampling.

The approach effectively transfers information across different ODDs and functionalities.

Proposes a stopping criterion based on information gain for evaluation campaigns.

Abstract

As autonomous vehicles (AVs) take on growing Operational Design Domains (ODDs), they need to go through a systematic, transparent, and scalable evaluation process to demonstrate their benefits to society. Current scenario sampling techniques for AV performance evaluation usually focus on a specific functionality, such as lane changing, and do not accommodate a transfer of information about an AV system from one ODD to the next. In this paper, we reformulate the scenario sampling problem across ODDs and functionalities as a submodular optimization problem. To do so, we abstract AV performance as a Bayesian Hierarchical Model, which we use to infer information gained by revealing performance in new scenarios. We propose the information gain as a measure of scenario relevance and evaluation progress. Furthermore, we leverage the submodularity, or diminishing returns, property of the information gain not only to find a near-optimal scenario set, but also to propose a stopping criterion for an AV performance evaluation campaign. We find that we only need to explore about 7.5% of the scenario space to meet this criterion, a 23% improvement over Latin Hypercube Sampling.