What's Making That Sound Right Now? Video-centric Audio-Visual Localization

TL;DR

This paper introduces AVATAR, a new benchmark with temporal scenarios for audio-visual localization, and TAVLO, a model that effectively uses temporal information to improve source localization accuracy.

Contribution

The paper proposes AVATAR, a comprehensive video-centric AVL benchmark, and TAVLO, a novel model that explicitly incorporates high-resolution temporal information for better localization.

Findings

TAVLO outperforms conventional methods in temporal tracking.

Temporal dynamics are crucial for accurate audio-visual localization.

AVATAR enables more realistic evaluation of AVL models.

Abstract

Audio-Visual Localization (AVL) aims to identify sound-emitting sources within a visual scene. However, existing studies focus on image-level audio-visual associations, failing to capture temporal dynamics. Moreover, they assume simplified scenarios where sound sources are always visible and involve only a single object. To address these limitations, we propose AVATAR, a video-centric AVL benchmark that incorporates high-resolution temporal information. AVATAR introduces four distinct scenarios -- Single-sound, Mixed-sound, Multi-entity, and Off-screen -- enabling a more comprehensive evaluation of AVL models. Additionally, we present TAVLO, a novel video-centric AVL model that explicitly integrates temporal information. Experimental results show that conventional methods struggle to track temporal variations due to their reliance on global audio features and frame-level mappings. In contrast, TAVLO achieves robust and precise audio-visual alignment by leveraging high-resolution temporal modeling. Our work empirically demonstrates the importance of temporal dynamics in AVL and establishes a new standard for video-centric audio-visual localization.