RoboDriveVLM: A Novel Benchmark and Baseline towards Robust Vision-Language Models for Autonomous Driving

TL;DR

This paper introduces RoboDriveBench, a comprehensive robustness benchmark for vision-language models in autonomous driving, and proposes RoboDriveVLM, a new framework with test-time adaptation to enhance system reliability under real-world challenges.

Contribution

It presents RoboDriveBench for evaluating VLM robustness in autonomous driving and introduces RoboDriveVLM with multimodal data integration and test-time adaptation techniques.

Findings

Current VLM systems are vulnerable to real-world sensor and prompt corruptions.

RoboDriveVLM improves robustness through multimodal data mapping.

Test-time adaptation enhances VLM performance under diverse challenges.

Abstract

Current Vision-Language Model (VLM)-based end-to-end autonomous driving systems often leverage large language models to generate driving decisions directly based on their understanding of the current scene. However, such systems introduce multiple risks in real-world driving scenarios. To evaluate whether VLMs are truly viable for autonomous driving, we introduce RoboDriveBench, the first robustness benchmark focused on end-to-end trajectory prediction tasks. This benchmark systematically evaluates two critical categories of real-world challenges for VLM-based end-to-end autonomous driving systems through 11 simulated scenarios encompassing various corruption types, including 6 scenarios of sensor corruption caused by environmental variations, along with 5 cases of prompt corruption resulting from human intervention and data transmission failures. Each corruption type includes 250 unique driving scenarios and 5,689 frames, resulting in 64,559 total trajectory prediction cases per evaluation. To overcome these real-world challenges, we propose a novel VLM-based autonomous driving framework called RoboDriveVLM, which enhances robustness by mapping more multimodal data-e.g., lidar and radar-into a unified latent space. Furthermore, we introduce a new Test-Time Adaptation (TTA) method based on cross-modal knowledge distillation to improve the robustness of VLM-based autonomous driving systems. Through extensive experiments, our work highlights the limitations of current VLM-based end-to-end autonomous driving systems and provides a more reliable solution for real-world deployment. Source code and datasets will be released.