Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation

TL;DR

This paper introduces AuTR, a multimodal transformer architecture that improves audio-visual segmentation by enabling deep feature fusion and focusing on sounding objects using audio cues, outperforming previous methods.

Contribution

The paper presents a novel transformer-based model with an audio-aware query mechanism for enhanced audio-visual segmentation performance.

Findings

Outperforms previous AVS methods in accuracy.

Demonstrates better generalization in multi-sound scenarios.

Effectively focuses on sounding objects while ignoring silent ones.

Abstract

The goal of the audio-visual segmentation (AVS) task is to segment the sounding objects in the video frames using audio cues. However, current fusion-based methods have the performance limitations due to the small receptive field of convolution and inadequate fusion of audio-visual features. To overcome these issues, we propose a novel \textbf{Au}dio-aware query-enhanced \textbf{TR}ansformer (AuTR) to tackle the task. Unlike existing methods, our approach introduces a multimodal transformer architecture that enables deep fusion and aggregation of audio-visual features. Furthermore, we devise an audio-aware query-enhanced transformer decoder that explicitly helps the model focus on the segmentation of the pinpointed sounding objects based on audio signals, while disregarding silent yet salient objects. Experimental results show that our method outperforms previous methods and demonstrates better generalization ability in multi-sound and open-set scenarios.