LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving

TL;DR

This paper introduces LMAD, a novel vision-language framework for autonomous driving that enhances scene understanding and explainability by integrating specialized adapters and comprehensive scene reasoning, improving driving reasoning performance.

Contribution

The paper presents a new end-to-end vision-language model tailored for autonomous driving, incorporating scene interaction and expert adapters for better spatial awareness and explainability.

Findings

Significantly improves performance on driving reasoning tasks

Sets new standards in explainable autonomous driving

Compatible with existing vision-language models

Abstract

Large vision-language models (VLMs) have shown promising capabilities in scene understanding, enhancing the explainability of driving behaviors and interactivity with users. Existing methods primarily fine-tune VLMs on on-board multi-view images and scene reasoning text, but this approach often lacks the holistic and nuanced scene recognition and powerful spatial awareness required for autonomous driving, especially in complex situations. To address this gap, we propose a novel vision-language framework tailored for autonomous driving, called LMAD. Our framework emulates modern end-to-end driving paradigms by incorporating comprehensive scene understanding and a task-specialized structure with VLMs. In particular, we introduce preliminary scene interaction and specialized expert adapters within the same driving task structure, which better align VLMs with autonomous driving scenarios. Furthermore, our approach is designed to be fully compatible with existing VLMs while seamlessly integrating with planning-oriented driving systems. Extensive experiments on the DriveLM and nuScenes-QA datasets demonstrate that LMAD significantly boosts the performance of existing VLMs on driving reasoning tasks,setting a new standard in explainable autonomous driving.