Weakly-Supervised Audio-Visual Segmentation

TL;DR

This paper introduces WS-AVS, a weakly-supervised framework for audio-visual segmentation that uses instance-level annotations and multi-scale contrastive learning to effectively segment sound sources in videos.

Contribution

The paper proposes a novel weakly-supervised framework, WS-AVS, that leverages multi-scale contrastive learning for audio-visual segmentation with less detailed supervision.

Findings

WS-AVS outperforms existing methods on AVSBench.

Effective in both single-source and multi-source scenarios.

Reduces reliance on pixel-wise masks.

Abstract

Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e., weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.