ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving

TL;DR

ViLaD introduces a diffusion-based vision language framework for autonomous driving that enables faster, bidirectional decision-making, outperforming traditional autoregressive models in accuracy and speed, and demonstrating real-world applicability.

Contribution

The paper presents ViLaD, a novel diffusion model architecture for autonomous driving that reduces latency and enables bidirectional reasoning, addressing limitations of autoregressive models.

Findings

Outperforms state-of-the-art autoregressive models in planning accuracy.

Achieves significantly faster inference speed.

Demonstrates practical deployment on an autonomous vehicle.

Abstract

End-to-end autonomous driving systems built on Vision Language Models (VLMs) have shown significant promise, yet their reliance on autoregressive architectures introduces some limitations for real-world applications. The sequential, token-by-token generation process of these models results in high inference latency and cannot perform bidirectional reasoning, making them unsuitable for dynamic, safety-critical environments. To overcome these challenges, we introduce ViLaD, a novel Large Vision Language Diffusion (LVLD) framework for end-to-end autonomous driving that represents a paradigm shift. ViLaD leverages a masked diffusion model that enables parallel generation of entire driving decision sequences, significantly reducing computational latency. Moreover, its architecture supports bidirectional reasoning, allowing the model to consider both past and future simultaneously, and supports progressive easy-first generation to iteratively improve decision quality. We conduct comprehensive experiments on the nuScenes dataset, where ViLaD outperforms state-of-the-art autoregressive VLM baselines in both planning accuracy and inference speed, while achieving a near-zero failure rate. Furthermore, we demonstrate the framework's practical viability through a real-world deployment on an autonomous vehicle for an interactive parking task, confirming its effectiveness and soundness for practical applications.