A Hierarchical Test Platform for Vision Language Model (VLM)-Integrated Real-World Autonomous Driving

TL;DR

This paper presents a hierarchical real-world testing platform for evaluating vision-language model-integrated autonomous vehicles, addressing domain shift challenges and enabling flexible, authentic scenario testing in real-world conditions.

Contribution

It introduces a modular, low-latency middleware and a configurable testing environment for comprehensive evaluation of VLM-based autonomous driving systems.

Findings

Effective testing of VLM-enabled vehicles demonstrated

Supports diverse real-world scenarios and conditions

Facilitates robust experimentation and evaluation

Abstract

Vision-Language Models (VLMs) have demonstrated notable promise in autonomous driving by offering the potential for multimodal reasoning through pretraining on extensive image-text pairs. However, adapting these models from broad web-scale data to the safety-critical context of driving presents a significant challenge, commonly referred to as domain shift. Existing simulation-based and dataset-driven evaluation methods, although valuable, often fail to capture the full complexity of real-world scenarios and cannot easily accommodate repeatable closed-loop testing with flexible scenario manipulation. In this paper, we introduce a hierarchical real-world test platform specifically designed to evaluate VLM-integrated autonomous driving systems. Our approach includes a modular, low-latency on-vehicle middleware that allows seamless incorporation of various VLMs, a clearly separated perception-planning-control architecture that can accommodate both VLM-based and conventional modules, and a configurable suite of real-world testing scenarios on a closed track that facilitates controlled yet authentic evaluations. We demonstrate the effectiveness of the proposed platform`s testing and evaluation ability with a case study involving a VLM-enabled autonomous vehicle, highlighting how our test framework supports robust experimentation under diverse conditions.