Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

TL;DR

This paper introduces Bench2Drive-VL, a comprehensive closed-loop benchmark for vision-language models in autonomous driving, enabling more realistic evaluation of model performance in diverse driving scenarios.

Contribution

It extends existing benchmarks by providing a closed-loop evaluation environment with diverse, behavior-grounded questions, and a flexible framework for VLMs in autonomous driving.

Findings

Enables evaluation of VLMs under out-of-distribution driving scenarios.

Provides a unified interface for integrating VLMs into closed-loop driving simulation.

Open sources code and datasets for community use.

Abstract

With the rise of vision-language models (VLM), their application for autonomous driving (VLM4AD) has gained significant attention. Meanwhile, in autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation, as it can evaluate the performance of the model under cumulative errors and out-of-distribution inputs. However, existing VLM4AD benchmarks evaluate the model`s scene understanding ability under open-loop, i.e., via static question-answer (QA) dataset. This kind of evaluation fails to assess the VLMs performance under out-of-distribution states rarely appeared in the human collected datasets.To this end, we present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving, which introduces: (1) DriveCommenter, a closed-loop generator that automatically generates diverse, behavior-grounded question-answer pairs for all driving situations in CARLA,including severe off-route and off-road deviations previously unassessable in simulation. (2) A unified protocol and interface that allows modern VLMs to be directly plugged into the Bench2Drive closed-loop environment to compare with traditional agents. (3) A flexible reasoning and control framework, supporting multi-format visual inputs and configurable graph-based chain-of-thought execution. (4) A complete development ecosystem. Together, these components form a comprehensive closed-loop benchmark for VLM4AD. All codes and annotated datasets are open sourced.