Revisiting Audio-Visual Segmentation with Vision-Centric Transformer

TL;DR

This paper introduces a vision-centric Transformer framework for audio-visual segmentation that improves object distinction and contour accuracy by leveraging vision-derived queries and prototype prompting, achieving state-of-the-art results.

Contribution

The paper proposes a novel vision-centric Transformer with a prototype prompted query generation module for enhanced audio-visual segmentation.

Findings

Achieves new state-of-the-art performance on AVSBench datasets.

Effectively distinguishes sound-producing objects from mixed audio.

Improves contour delineation accuracy in video segmentation.

Abstract

Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where object queries are derived from audio features. However, audio-centric Transformers suffer from two limitations: perception ambiguity caused by the mixed nature of audio, and weakened dense prediction ability due to visual detail loss. To address these limitations, we propose a new Vision-Centric Transformer (VCT) framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information, enabling queries to better distinguish between different sounding objects from mixed audio and accurately delineate their contours. Additionally, we also introduce a Prototype Prompted Query Generation (PPQG) module within our VCT framework to generate vision-derived queries that are both semantically aware and visually rich through audio prototype prompting and pixel context grouping, facilitating audio-visual information aggregation. Extensive experiments demonstrate that our VCT framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset. The code is available at https://github.com/spyflying/VCT_AVS.