Look and Tell: A Dataset for Multimodal Grounding Across Egocentric and Exocentric Views

TL;DR

Look and Tell is a multimodal dataset capturing synchronized gaze, speech, and video from egocentric and exocentric views, enabling research on spatial grounding and situated dialogue in embodied agents.

Contribution

The paper introduces a novel dataset combining egocentric and exocentric perspectives with multimodal annotations for studying spatial grounding in communication.

Findings

Provides a new benchmark for multimodal grounding across perspectives

Includes extensive annotations of referential expressions

Facilitates research on embodied dialogue understanding

Abstract

We introduce Look and Tell, a multimodal dataset for studying referential communication across egocentric and exocentric perspectives. Using Meta Project Aria smart glasses and stationary cameras, we recorded synchronized gaze, speech, and video as 25 participants instructed a partner to identify ingredients in a kitchen. Combined with 3D scene reconstructions, this setup provides a benchmark for evaluating how different spatial representations (2D vs. 3D; ego vs. exo) affect multimodal grounding. The dataset contains 3.67 hours of recordings, including 2,707 richly annotated referential expressions, and is designed to advance the development of embodied agents that can understand and engage in situated dialogue.