Audio-Visual Segmentation

TL;DR

This paper introduces the novel task of audio-visual segmentation, creates a benchmark dataset, and proposes a new method that integrates audio and visual information for pixel-level object segmentation in videos.

Contribution

It defines the AVS problem, constructs the first AVS benchmark dataset, and proposes a novel audio-visual interaction method for segmentation.

Findings

The proposed method outperforms existing related methods.

The AVSBench dataset provides detailed pixel-wise annotations.

The approach effectively links audio semantics with visual segmentation.

Abstract

We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise annotations for the sounding objects in audible videos. Two settings are studied with this benchmark: 1) semi-supervised audio-visual segmentation with a single sound source and 2) fully-supervised audio-visual segmentation with multiple sound sources. To deal with the AVS problem, we propose a novel method that uses a temporal pixel-wise audio-visual interaction module to inject audio semantics as guidance for the visual segmentation process. We also design a regularization loss to encourage the audio-visual mapping during training. Quantitative and qualitative experiments on the AVSBench compare our approach to several existing methods from related tasks, demonstrating that the proposed method is promising for building a bridge between the audio and pixel-wise visual semantics. Code is available at https://github.com/OpenNLPLab/AVSBench.