Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios

TL;DR

This paper evaluates various probabilistic metamodels for efficiently characterizing complex driving scenarios in automated vehicle testing, emphasizing the importance of test case selection over model choice.

Contribution

It introduces an iterative test case selection approach and compares the predictive performance of different metamodels for AV scenario analysis.

Findings

Test case selection impacts predictive accuracy more than model choice.

Bayesian neural networks excel with large data and complex scenarios.

Gaussian processes offer higher reliability for simpler models.

Abstract

To validate the safety of automated vehicles (AV), scenario-based testing aims to systematically describe driving scenarios an AV might encounter. In this process, continuous inputs such as velocities result in an infinite number of possible variations of a scenario. Thus, metamodels are used to perform analyses or to select specific variations for examination. However, despite the safety criticality of AV testing, metamodels are usually seen as a part of an overall approach, and their predictions are not questioned. This paper analyzes the predictive performance of Gaussian processes (GP), deep Gaussian processes, extra-trees, and Bayesian neural networks (BNN), considering four scenarios with 5 to 20 inputs. Building on this, an iterative approach is introduced and evaluated, which allows to efficiently select test cases for common analysis tasks. The results show that regarding predictive performance, the appropriate selection of test cases is more important than the choice of metamodels. However, the choice of metamodels remains crucial: Their great flexibility allows BNNs to benefit from large amounts of data and to model even the most complex scenarios. In contrast, less flexible models like GPs convince with higher reliability. Hence, relevant test cases are best explored using scalable virtual test setups and flexible models. Subsequently, more realistic test setups and more reliable models can be used for targeted testing and validation.