Learning Vision-Language-Action World Models for Autonomous Driving

TL;DR

VLA-World introduces a unified vision-language-action world model that enhances autonomous driving by combining predictive imagination with reflective reasoning, leading to improved foresight and safety.

Contribution

The paper presents VLA-World, a novel VLA world model that integrates future scene generation with reasoning, supported by a new dataset and a three-stage training process.

Findings

VLA-World outperforms existing models on planning benchmarks.

The model achieves higher accuracy in future scene generation.

Reflective reasoning improves trajectory prediction quality.

Abstract

Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving by integrating perception, reasoning, and control within a unified multimodal framework. However, they often lack explicit modeling of temporal dynamics and global world consistency, which limits their foresight and safety. In contrast, world models can simulate plausible future scenes but generally struggle to reason about or evaluate the imagined future they generate. In this work, we present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning to improve driving foresight. VLA-World first uses an action-derived feasible trajectory to guide the generation of the next-frame image, capturing rich spatial and temporal cues that describe how the surrounding environment evolves. The model then reasons over this self-generated future imagined frame to refine the predicted trajectory, achieving higher performance and better interpretability. To support this pipeline, we curate nuScenes-GR-20K, a generative reasoning dataset derived from nuScenes, and employ a three-stage training strategy that includes pretraining, supervised fine-tuning, and reinforcement learning. Extensive experiments demonstrate that VLA-World consistently surpasses state-of-the-art VLA and world-model baselines on both planning and future-generation benchmarks. Project page: https://vlaworld.github.io