VLAD: A VLM-Augmented Autonomous Driving Framework with Hierarchical Planning and Interpretable Decision Process

TL;DR

This paper introduces VLAD, a novel autonomous driving framework that integrates vision-language models with hierarchical planning and interpretable decision-making, significantly improving safety and transparency in real-world scenarios.

Contribution

The paper presents a new VLM-augmented autonomous driving system with specialized fine-tuning, high-level command generation, and natural language explanations, advancing interpretability and performance.

Findings

Reduces collision rates by 31.82% on nuScenes dataset

Enhances spatial reasoning through custom question-answer training

Establishes new benchmark for VLM-based autonomous driving

Abstract

Recent advancements in open-source Visual Language Models (VLMs) such as LLaVA, Qwen-VL, and Llama have catalyzed extensive research on their integration with diverse systems. The internet-scale general knowledge encapsulated within these models presents significant opportunities for enhancing autonomous driving perception, prediction, and planning capabilities. In this paper we propose VLAD, a vision-language autonomous driving model, which integrates a fine-tuned VLM with VAD, a state-of-the-art end-to-end system. We implement a specialized fine-tuning approach using custom question-answer datasets designed specifically to improve the spatial reasoning capabilities of the model. The enhanced VLM generates high-level navigational commands that VAD subsequently processes to guide vehicle operation. Additionally, our system produces interpretable natural language explanations of driving decisions, thereby increasing transparency and trustworthiness of the traditionally black-box end-to-end architecture. Comprehensive evaluation on the real-world nuScenes dataset demonstrates that our integrated system reduces average collision rates by 31.82% compared to baseline methodologies, establishing a new benchmark for VLM-augmented autonomous driving systems.