BridgeEQA: Virtual Embodied Agents for Real Bridge Inspections

TL;DR

BridgeEQA introduces a new benchmark and evaluation metric for embodied agents performing real-world bridge inspections, emphasizing multi-scale reasoning and long-range spatial understanding.

Contribution

The paper presents BridgeEQA, a novel benchmark with professional inspection reports and a new metric, along with EMVR, a method that improves embodied question answering in infrastructure inspection scenarios.

Findings

State-of-the-art models show significant performance gaps on BridgeEQA.

EMVR outperforms baseline models in the inspection EQA task.

BridgeEQA provides a standardized platform for evaluating embodied agents in real-world inspections.

Abstract

Deploying embodied agents that can answer questions about their surroundings in realistic real-world settings remains difficult, partly due to the scarcity of benchmarks for episodic memory Embodied Question Answering (EQA). Inspired by the challenges of infrastructure inspections, we propose Inspection EQA as a compelling problem class for advancing episodic memory EQA. It demands multi-scale reasoning and long-range spatial understanding, while offering standardized evaluation, professional inspection reports as grounding, and egocentric imagery. We introduce BridgeEQA, a benchmark of 2,200 open-vocabulary question-answer pairs (in the style of OpenEQA) grounded in professional inspection reports across 200 real-world bridge scenes with 47.93 images on average per scene. We further propose a new EQA metric Image Citation Relevance to evaluate the ability of a model to cite relevant images. Evaluations of state-of-the-art vision-language models reveal substantial performance gaps. To address this, we propose Embodied Memory Visual Reasoning (EMVR), which formulates the inspection EQA task as a Markov decision process. EMVR shows strong performance over the baselines. Code and dataset are available at https://drags99.github.io/bridge-eqa/