Generative Scenario Rollouts for End-to-End Autonomous Driving

TL;DR

This paper introduces GeRo, a generative framework for autonomous driving that uses language-grounded scenario rollouts to improve planning, stability, and interpretability in complex driving environments.

Contribution

The paper presents a novel generative approach with language-conditioned scenario rollouts, enhancing long-horizon reasoning and multi-agent planning in autonomous driving systems.

Findings

Improves driving score and success rate significantly on Bench2Drive.

Achieves state-of-the-art performance with reinforcement learning integration.

Demonstrates strong zero-shot robustness in diverse scenarios.

Abstract

Vision-Language-Action (VLA) models are emerging as highly effective planning models for end-to-end autonomous driving systems. However, current works mostly rely on imitation learning from sparse trajectory annotations and under-utilize their potential as generative models. We propose Generative Scenario Rollouts (GeRo), a plug-and-play framework for VLA models that jointly performs planning and generation of language-grounded future traffic scenes through an autoregressive rollout strategy. First, a VLA model is trained to encode ego vehicle and agent dynamics into latent tokens under supervision from planning, motion, and language tasks, facilitating text-aligned generation. Next, GeRo performs language-conditioned autoregressive generation. Given multi-view images, a scenario description, and ego-action questions, it generates future latent tokens and textual responses to guide long-horizon rollouts. A rollout-consistency loss stabilizes predictions using ground truth or pseudo-labels, mitigating drift and preserving text-action alignment. This design enables GeRo to perform temporally consistent, language-grounded rollouts that support long-horizon reasoning and multi-agent planning. On Bench2Drive, GeRo improves driving score and success rate by +15.7 and +26.2, respectively. By integrating reinforcement learning with generative rollouts, GeRo achieves state-of-the-art closed-loop and open-loop performance, demonstrating strong zero-shot robustness. These results highlight the promise of generative, language-conditioned reasoning as a foundation for safer and more interpretable end-to-end autonomous driving.