Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future

TL;DR

This paper reviews the evolution of Vision-Language-Action models in autonomous driving, highlighting their potential for more interpretable, robust, and human-aligned driving policies by integrating perception, reasoning, and language-grounded decision making.

Contribution

It provides a structured overview of VLA frameworks, categorizes existing methods, and discusses key challenges and future directions in autonomous driving research.

Findings

VLA models unify perception and decision-making for better interpretability.

Two main paradigms: End-to-End VLA and Dual-System VLA.

Identifies challenges like robustness, interpretability, and instruction fidelity.

Abstract

Autonomous driving has long relied on modular "Perception-Decision-Action" pipelines, where hand-crafted interfaces and rule-based components often break down in complex or long-tailed scenarios. Their cascaded design further propagates perception errors, degrading downstream planning and control. Vision-Action (VA) models address some limitations by learning direct mappings from visual inputs to actions, but they remain opaque, sensitive to distribution shifts, and lack structured reasoning or instruction-following capabilities. Recent progress in Large Language Models (LLMs) and multimodal learning has motivated the emergence of Vision-Language-Action (VLA) frameworks, which integrate perception with language-grounded decision making. By unifying visual understanding, linguistic reasoning, and actionable outputs, VLAs offer a pathway toward more interpretable, generalizable, and human-aligned driving policies. This work provides a structured characterization of the emerging VLA landscape for autonomous driving. We trace the evolution from early VA approaches to modern VLA frameworks and organize existing methods into two principal paradigms: End-to-End VLA, which integrates perception, reasoning, and planning within a single model, and Dual-System VLA, which separates slow deliberation (via VLMs) from fast, safety-critical execution (via planners). Within these paradigms, we further distinguish subclasses such as textual vs. numerical action generators and explicit vs. implicit guidance mechanisms. We also summarize representative datasets and benchmarks for evaluating VLA-based driving systems and highlight key challenges and open directions, including robustness, interpretability, and instruction fidelity. Overall, this work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.