Improving Audio-Visual Segmentation with Bidirectional Generation

TL;DR

This paper introduces a bidirectional generation framework for audio-visual segmentation that models the correlation between visual features and sounds, improving accuracy especially in complex scenes with multiple sound sources.

Contribution

It proposes a novel bidirectional generation approach with visual-to-audio projection and volumetric motion estimation, advancing AVS performance on benchmark datasets.

Findings

Achieved new state-of-the-art results on AVSBench, especially in MS3 subset.

Demonstrated the effectiveness of bidirectional modeling in AVS.

Enhanced handling of temporal dynamics with motion estimation module.

Abstract

The aim of audio-visual segmentation (AVS) is to precisely differentiate audible objects within videos down to the pixel level. Traditional approaches often tackle this challenge by combining information from various modalities, where the contribution of each modality is implicitly or explicitly modeled. Nevertheless, the interconnections between different modalities tend to be overlooked in audio-visual modeling. In this paper, inspired by the human ability to mentally simulate the sound of an object and its visual appearance, we introduce a bidirectional generation framework. This framework establishes robust correlations between an object's visual characteristics and its associated sound, thereby enhancing the performance of AVS. To achieve this, we employ a visual-to-audio projection component that reconstructs audio features from object segmentation masks and minimizes reconstruction errors. Moreover, recognizing that many sounds are linked to object movements, we introduce an implicit volumetric motion estimation module to handle temporal dynamics that may be challenging to capture using conventional optical flow methods. To showcase the effectiveness of our approach, we conduct comprehensive experiments and analyses on the widely recognized AVSBench benchmark. As a result, we establish a new state-of-the-art performance level in the AVS benchmark, particularly excelling in the challenging MS3 subset which involves segmenting multiple sound sources. To facilitate reproducibility, we plan to release both the source code and the pre-trained model.