Box-QAymo: Box-Referring VQA Dataset for Autonomous Driving

TL;DR

Box-QAymo introduces a novel dataset and benchmark for evaluating vision-language models in autonomous driving, focusing on localized, user-driven queries involving spatial and temporal reasoning about objects.

Contribution

The paper presents a new box-referring dataset and hierarchical evaluation protocol tailored for autonomous driving scenarios, enabling assessment and fine-tuning of models on complex spatial-temporal reasoning tasks.

Findings

Current VLMs show significant limitations in perception-based questions.

The dataset captures diverse object classes and attributes relevant to driving.

Evaluation reveals gaps in models' ability to understand and reason about dynamic scenes.

Abstract

Interpretable communication is essential for safe and trustworthy autonomous driving, yet current vision-language models (VLMs) often operate under idealized assumptions and struggle to capture user intent in real-world scenarios. Existing driving-oriented VQA datasets are limited to full-scene descriptions or waypoint prediction, preventing the assessment of whether VLMs can respond to localized user-driven queries. We introduce Box-QAymo, a box-referring dataset and benchmark designed to both evaluate and finetune VLMs on spatial and temporal reasoning over user-specified objects. Users express intent by drawing bounding boxes, offering a fast and intuitive interface for focused queries in complex scenes. Specifically, we propose a hierarchical evaluation protocol that begins with binary sanity-check questions to assess basic model capacities, and progresses to (1) attribute prediction for box-referred objects, (2) motion understanding of target instances, and (3) spatiotemporal motion reasoning over inter-object dynamics across frames. To support this, we crowd-sourced fine-grained object classes and visual attributes that reflect the complexity drivers encounter, and extract object trajectories to construct temporally grounded QA pairs. Rigorous quality control through negative sampling, temporal consistency checks, and difficulty-aware balancing guarantee dataset robustness and diversity. Our comprehensive evaluation reveals significant limitations in current VLMs when queried about perception questions, highlighting the gap in achieving real-world performance. This work provides a foundation for developing more robust and interpretable autonomous driving systems that can communicate effectively with users under real-world conditions. Project page and dataset are available at https://djamahl99.github.io/qaymo-pages/.