Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localization

TL;DR

This paper introduces a novel audio-visual spatial integration and recursive attention network that mimics human behavior to improve the accuracy and robustness of sound source localization in visual scenes.

Contribution

It proposes a new network architecture combining spatial cue integration and recursive attention, along with specialized loss functions, to enhance sound source localization performance.

Findings

Outperforms existing methods on Flickr SoundNet and VGG-Sound Source datasets.

Demonstrates improved localization accuracy and robustness.

Validates effectiveness of recursive attention and spatial integration components.

Abstract

The objective of the sound source localization task is to enable machines to detect the location of sound-making objects within a visual scene. While the audio modality provides spatial cues to locate the sound source, existing approaches only use audio as an auxiliary role to compare spatial regions of the visual modality. Humans, on the other hand, utilize both audio and visual modalities as spatial cues to locate sound sources. In this paper, we propose an audio-visual spatial integration network that integrates spatial cues from both modalities to mimic human behavior when detecting sound-making objects. Additionally, we introduce a recursive attention network to mimic human behavior of iterative focusing on objects, resulting in more accurate attention regions. To effectively encode spatial information from both modalities, we propose audio-visual pair matching loss and spatial region alignment loss. By utilizing the spatial cues of audio-visual modalities and recursively focusing objects, our method can perform more robust sound source localization. Comprehensive experimental results on the Flickr SoundNet and VGG-Sound Source datasets demonstrate the superiority of our proposed method over existing approaches. Our code is available at: https://github.com/VisualAIKHU/SIRA-SSL