AutoDriDM: An Explainable Benchmark for Decision-Making of Vision-Language Models in Autonomous Driving

TL;DR

AutoDriDM introduces a comprehensive, decision-focused benchmark for evaluating vision-language models in autonomous driving, emphasizing reasoning and decision-making over perception alone.

Contribution

The paper presents AutoDriDM, a novel benchmark with 6,650 questions across Object, Scene, and Decision dimensions, and analyzes perception-decision boundaries and reasoning failures in VLMs.

Findings

Weak correlation between perception and decision performance.

Identification of key failure modes like logical reasoning errors.

Introduction of an automated annotation analyzer model.

Abstract

Autonomous driving is a highly challenging domain that requires reliable perception and safe decision-making in complex scenarios. Recent vision-language models (VLMs) demonstrate reasoning and generalization abilities, opening new possibilities for autonomous driving; however, existing benchmarks and metrics overemphasize perceptual competence and fail to adequately assess decision-making processes. In this work, we present AutoDriDM, a decision-centric, progressive benchmark with 6,650 questions across three dimensions - Object, Scene, and Decision. We evaluate mainstream VLMs to delineate the perception-to-decision capability boundary in autonomous driving, and our correlation analysis reveals weak alignment between perception and decision-making performance. We further conduct explainability analyses of models' reasoning processes, identifying key failure modes such as logical reasoning errors, and introduce an analyzer model to automate large-scale annotation. AutoDriDM bridges the gap between perception-centered and decision-centered evaluation, providing guidance toward safer and more reliable VLMs for real-world autonomous driving.