Driving scenario generation and evaluation using a structured layer representation and foundational models

TL;DR

This paper introduces a structured five-layer model for generating and evaluating rare driving scenarios for autonomous vehicles, leveraging foundational models and new metrics to improve scenario diversity and relevance.

Contribution

It proposes a novel layered scenario representation with subclasses and characteristics, enhancing generation and evaluation of synthetic driving scenarios.

Findings

The structured layer model improves scenario comparison and diversity assessment.

The adapted metrics effectively evaluate the relevance of synthetic datasets.

Qualitative analysis shows realistic synthetic videos from structured descriptions.

Abstract

Rare and challenging driving scenarios are critical for autonomous vehicle development. Since they are difficult to encounter, simulating or generating them using generative models is a popular approach. Following previous efforts to structure driving scenario representations in a layer model, we propose a structured five-layer model to improve the evaluation and generation of rare scenarios. We use this model alongside large foundational models to generate new driving scenarios using a data augmentation strategy. Unlike previous representations, our structure introduces subclasses and characteristics for every agent of the scenario, allowing us to compare them using an embedding specific to our layer-model. We study and adapt two metrics to evaluate the relevance of a synthetic dataset in the context of a structured representation: the diversity score estimates how different the scenarios of a dataset are from one another, while the originality score calculates how similar a synthetic dataset is from a real reference set. This paper showcases both metrics in different generation setup, as well as a qualitative evaluation of synthetic videos generated from structured scenario descriptions. The code and extended results can be found at https://github.com/Valgiz/5LMSG.