Transavs: End-To-End Audio-Visual Segmentation With Transformer

TL;DR

TransAVS introduces a transformer-based end-to-end framework for audio-visual segmentation, effectively disentangling audio signals and improving segmentation accuracy by leveraging self-supervised learning.

Contribution

It is the first to apply transformer architecture to AVS, explicitly disentangles audio streams as queries, and employs self-supervised losses for better object distinction.

Findings

Achieves state-of-the-art results on AVSBench dataset.

Effectively disentangles audio signals for clearer segmentation.

Improves distinction between similar sounding objects.

Abstract

Audio-Visual Segmentation (AVS) is a challenging task, which aims to segment sounding objects in video frames by exploring audio signals. Generally AVS faces two key challenges: (1) Audio signals inherently exhibit a high degree of information density, as sounds produced by multiple objects are entangled within the same audio stream; (2) Objects of the same category tend to produce similar audio signals, making it difficult to distinguish between them and thus leading to unclear segmentation results. Toward this end, we propose TransAVS, the first Transformer-based end-to-end framework for AVS task. Specifically, TransAVS disentangles the audio stream as audio queries, which will interact with images and decode into segmentation masks with full transformer architectures. This scheme not only promotes comprehensive audio-image communication but also explicitly excavates instance cues encapsulated in the scene. Meanwhile, to encourage these audio queries to capture distinctive sounding objects instead of degrading to be homogeneous, we devise two self-supervised loss functions at both query and mask levels, allowing the model to capture distinctive features within similar audio data and achieve more precise segmentation. Our experiments demonstrate that TransAVS achieves state-of-the-art results on the AVSBench dataset, highlighting its effectiveness in bridging the gap between audio and visual modalities.