Unsupervised Audio-Visual Segmentation with Modality Alignment

TL;DR

This paper introduces an unsupervised audio-visual segmentation method called MoCA that leverages foundation models and contrastive learning to identify sound-producing objects at the pixel level without costly annotations.

Contribution

The paper presents a novel unsupervised learning framework, MoCA, integrating foundation models for multi-modality association in audio-visual segmentation, reducing reliance on annotated data.

Findings

MoCA outperforms baseline methods on AVSBench and AVSS datasets.

Achieves significant mIoU improvements, e.g., +17.24% on AVSBench S4.

Approaches supervised performance in complex multi-object scenarios.

Abstract

Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the object in a visual scene that produces a given sound. Current AVS methods rely on costly fine-grained annotations of mask-audio pairs, making them impractical for scalability. To address this, we introduce unsupervised AVS, eliminating the need for such expensive annotation. To tackle this more challenging problem, we propose an unsupervised learning method, named Modality Correspondence Alignment (MoCA), which seamlessly integrates off-the-shelf foundation models like DINO, SAM, and ImageBind. This approach leverages their knowledge complementarity and optimizes their joint usage for multi-modality association. Initially, we estimate positive and negative image pairs in the feature space. For pixel-level association, we introduce an audio-visual adapter and a novel pixel matching aggregation strategy within the image-level contrastive learning framework. This allows for a flexible connection between object appearance and audio signal at the pixel level, with tolerance to imaging variations such as translation and rotation. Extensive experiments on the AVSBench (single and multi-object splits) and AVSS datasets demonstrate that our MoCA outperforms strongly designed baseline methods and approaches supervised counterparts, particularly in complex scenarios with multiple auditory objects. Notably when comparing mIoU, MoCA achieves a substantial improvement over baselines in both the AVSBench (S4: +17.24%; MS3: +67.64%) and AVSS (+19.23%) audio-visual segmentation challenges.