EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?

TL;DR

EOC-Bench is a new benchmark for evaluating multimodal large language models' ability to understand, recall, and predict objects in dynamic egocentric environments, addressing a gap in existing static scene-focused benchmarks.

Contribution

The paper introduces EOC-Bench, a comprehensive benchmark with annotated QA pairs and a novel temporal accuracy metric for assessing object-centric cognition in dynamic egocentric scenarios.

Findings

MLLMs show varied performance across different temporal categories.

EOC-Bench enables systematic evaluation of object understanding in dynamic contexts.

Benchmark facilitates development of more reliable embodied cognition models.

Abstract

The emergence of multimodal large language models (MLLMs) has driven breakthroughs in egocentric vision applications. These applications necessitate persistent, context-aware understanding of objects, as users interact with tools in dynamic and cluttered environments. However, existing embodied benchmarks primarily focus on static scene exploration, emphasizing object's appearance and spatial attributes while neglecting the assessment of dynamic changes arising from users' interactions. To address this gap, we introduce EOC-Bench, an innovative benchmark designed to systematically evaluate object-centric embodied cognition in dynamic egocentric scenarios. Specially, EOC-Bench features 3,277 meticulously annotated QA pairs categorized into three temporal categories: Past, Present, and Future, covering 11 fine-grained evaluation dimensions and 3 visual object referencing types. To ensure thorough assessment, we develop a mixed-format human-in-the-loop annotation framework with four types of questions and design a novel multi-scale temporal accuracy metric for open-ended temporal evaluation. Based on EOC-Bench, we conduct comprehensive evaluations of various proprietary, open-source, and object-level MLLMs. EOC-Bench serves as a crucial tool for advancing the embodied object cognitive capabilities of MLLMs, establishing a robust foundation for developing reliable core models for embodied systems.