Multimodal Variational Auto-encoder based Audio-Visual Segmentation

TL;DR

This paper introduces ECMVAE, a novel multimodal variational auto-encoder for audio-visual segmentation that explicitly models shared and specific features of each modality, significantly improving performance on AVS tasks.

Contribution

The paper presents ECMVAE, a new explicit representation learning framework that factorizes modality representations and enhances cross-modal shared information for better AVS performance.

Findings

Achieved state-of-the-art results on AVSBench with a 3.84 mIOU improvement.

Effectively models shared and specific features of audio and visual data.

Demonstrated superior segmentation accuracy on the MS3 subset.

Abstract

We propose an Explicit Conditional Multimodal Variational Auto-Encoder (ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sources in the video sequence. Existing AVS methods focus on implicit feature fusion strategies, where models are trained to fit the discrete samples in the dataset. With a limited and less diverse dataset, the resulting performance is usually unsatisfactory. In contrast, we address this problem from an effective representation learning perspective, aiming to model the contribution of each modality explicitly. Specifically, we find that audio contains critical category information of the sound producers, and visual data provides candidate sound producer(s). Their shared information corresponds to the target sound producer(s) shown in the visual data. In this case, cross-modal shared representation learning is especially important for AVS. To achieve this, our ECMVAE factorizes the representations of each modality with a modality-shared representation and a modality-specific representation. An orthogonality constraint is applied between the shared and specific representations to maintain the exclusive attribute of the factorized latent code. Further, a mutual information maximization regularizer is introduced to achieve extensive exploration of each modality. Quantitative and qualitative evaluations on the AVSBench demonstrate the effectiveness of our approach, leading to a new state-of-the-art for AVS, with a 3.84 mIOU performance leap on the challenging MS3 subset for multiple sound source segmentation.