VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving

TL;DR

VLADriver-RAG introduces a retrieval-augmented vision-language-action framework for autonomous driving, leveraging explicit semantic graphs and a novel embedding model to improve trajectory planning and generalization.

Contribution

It proposes a structure-aware retrieval method using semantic graphs and Graph-DTW alignment, enhancing autonomous driving models' accuracy and robustness.

Findings

Achieved a new state-of-the-art Driving Score of 89.12 on Bench2Drive.

Effectively filters visual noise through semantic graph abstraction.

Utilizes a novel Scenario-Aligned Embedding Model for relevant retrieval.

Abstract

Vision-Language-Action (VLA) models have emerged as a promising paradigm for end-to-end autonomous driving, yet their reliance on implicit parametric knowledge limits generalization in long-tail scenarios. While Retrieval-Augmented Generation (RAG) offers a solution by accessing external expert priors, standard visual retrieval suffers from high latency and semantic ambiguity. To address these challenges, we propose \textbf{VLADriver-RAG}, a framework that grounds planning in explicit, structure-aware historical knowledge. Specifically, we abstract sensory inputs into spatiotemporal semantic graphs via a \textit{Visual-to-Scenario} mechanism, effectively filtering visual noise. To ensure retrieval relevance, we employ a \textit{Scenario-Aligned Embedding Model} that utilizes Graph-DTW metric alignment to prioritize intrinsic topological consistency over superficial visual similarity. These retrieved priors are then fused within a query-based VLA backbone to synthesize precise, disentangled trajectories. Extensive experiments on the Bench2Drive benchmark establish a new state-of-the-art, achieving a Driving Score of 89.12.