RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios

TL;DR

RealGen is a retrieval-augmented framework that enables controllable, flexible, and realistic traffic scenario generation for autonomous vehicle training by combining behaviors from multiple examples without relying on traditional memorization methods.

Contribution

It introduces a novel retrieval-based in-context learning approach for traffic scenario generation, allowing for scenario editing, composition, and control beyond existing methods.

Findings

RealGen achieves high flexibility in scenario generation.

It allows scenario editing and composition.

The method outperforms traditional dataset-memorization approaches.

Abstract

Simulation plays a crucial role in the development of autonomous vehicles (AVs) due to the potential risks associated with real-world testing. Although significant progress has been made in the visual aspects of simulators, generating complex behavior among agents remains a formidable challenge. It is not only imperative to ensure realism in the scenarios generated but also essential to incorporate preferences and conditions to facilitate controllable generation for AV training and evaluation. Traditional methods, mainly relying on memorizing the distribution of training datasets, often fall short in generating unseen scenarios. Inspired by the success of retrieval augmented generation in large language models, we present RealGen, a novel retrieval-based in-context learning framework for traffic scenario generation. RealGen synthesizes new scenarios by combining behaviors from multiple retrieved examples in a gradient-free way, which may originate from templates or tagged scenarios. This in-context learning framework endows versatile generative capabilities, including the ability to edit scenarios, compose various behaviors, and produce critical scenarios. Evaluations show that RealGen offers considerable flexibility and controllability, marking a new direction in the field of controllable traffic scenario generation. Check our project website for more information: https://realgen.github.io.