EgoSound: Benchmarking Sound Understanding in Egocentric Videos

TL;DR

EgoSound is a new benchmark designed to evaluate egocentric sound understanding in multimodal large language models, highlighting current models' strengths and limitations in multisensory perception.

Contribution

It introduces the first comprehensive benchmark for egocentric sound understanding, combining data from Ego4D and EgoBlind, with a seven-task taxonomy and extensive QA pairs.

Findings

Current models show emerging auditory reasoning abilities.

Models are limited in fine-grained spatial and causal understanding.

EgoSound provides a challenging foundation for multisensory egocentric intelligence.

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in vision-language understanding. Yet, human perception is inherently multisensory, integrating sight, sound, and motion to reason about the world. Among these modalities, sound provides indispensable cues about spatial layout, off-screen events, and causal interactions, particularly in egocentric settings where auditory and visual signals are tightly coupled. To this end, we introduce EgoSound, the first benchmark designed to systematically evaluate egocentric sound understanding in MLLMs. EgoSound unifies data from Ego4D and EgoBlind, encompassing both sighted and sound-dependent experiences. It defines a seven-task taxonomy spanning intrinsic sound perception, spatial localization, causal inference, and cross-modal reasoning. Constructed through a multi-stage auto-generative pipeline, EgoSound contains 7315 validated QA pairs across 900 videos. Comprehensive experiments on nine state-of-the-art MLLMs reveal that current models exhibit emerging auditory reasoning abilities but remain limited in fine-grained spatial and causal understanding. EgoSound establishes a challenging foundation for advancing multisensory egocentric intelligence, bridging the gap between seeing and truly hearing the world. Project page: https://groolegend.github.io/EgoSound/ .