Embodied Scene Understanding for Vision Language Models via MetaVQA

TL;DR

MetaVQA is a new benchmark for evaluating and improving vision language models' spatial reasoning and scene understanding in embodied AI tasks, especially for autonomous driving scenarios.

Contribution

We introduce MetaVQA, a comprehensive benchmark with question-answer pairs based on real-world traffic data to enhance VLMs' spatial reasoning and scene understanding capabilities.

Findings

Fine-tuning VLMs with MetaVQA improves spatial reasoning.

Enhanced VQA accuracy and safety-aware driving maneuvers.

Strong transferability from simulation to real-world observations.

Abstract

Vision Language Models (VLMs) demonstrate significant potential as embodied AI agents for various mobility applications. However, a standardized, closed-loop benchmark for evaluating their spatial reasoning and sequential decision-making capabilities is lacking. To address this, we present MetaVQA: a comprehensive benchmark designed to assess and enhance VLMs' understanding of spatial relationships and scene dynamics through Visual Question Answering (VQA) and closed-loop simulations. MetaVQA leverages Set-of-Mark prompting and top-down view ground-truth annotations from nuScenes and Waymo datasets to automatically generate extensive question-answer pairs based on diverse real-world traffic scenarios, ensuring object-centric and context-rich instructions. Our experiments show that fine-tuning VLMs with the MetaVQA dataset significantly improves their spatial reasoning and embodied scene comprehension in safety-critical simulations, evident not only in improved VQA accuracies but also in emerging safety-aware driving maneuvers. In addition, the learning demonstrates strong transferability from simulation to real-world observation. Code and data will be publicly available at https://metadriverse.github.io/metavqa .