AV-SSAN: Audio-Visual Selective DoA Estimation through Explicit Multi-Band Semantic-Spatial Alignment

TL;DR

This paper introduces AV-SSAN, a novel framework for audio-visual source localization that enables selective target localization using semantic prompts without requiring spatially paired data, and demonstrates its effectiveness on a new large-scale dataset.

Contribution

The paper proposes the AV-SSAN framework and MB-SSA Net for semantic-spatial alignment, addressing limitations of existing AV-SSL methods by enabling target-specific localization without spatial pairing.

Findings

Achieves 71.29% accuracy in target localization

Constructs the large-scale VGGSound-SSL dataset

Significantly outperforms existing AV-SSL methods

Abstract

Audio-visual sound source localization (AV-SSL) estimates the position of sound sources by fusing auditory and visual cues. Current AV-SSL methodologies typically require spatially-paired audio-visual data and cannot selectively localize specific target sources. To address these limitations, we introduce Cross-Instance Audio-Visual Localization (CI-AVL), a novel task that localizes target sound sources using visual prompts from different instances of the same semantic class. CI-AVL enables selective localization without spatially paired data. To solve this task, we propose AV-SSAN, a semantic-spatial alignment framework centered on a Multi-Band Semantic-Spatial Alignment Network (MB-SSA Net). MB-SSA Net decomposes the audio spectrogram into multiple frequency bands, aligns each band with semantic visual prompts, and refines spatial cues to estimate the direction-of-arrival (DoA). To facilitate this research, we construct VGGSound-SSL, a large-scale dataset comprising 13,981 spatial audio clips across 296 categories, each paired with visual prompts. AV-SSAN achieves a mean absolute error of 16.59 and an accuracy of 71.29%, significantly outperforming existing AV-SSL methods. Code and data will be public.