Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation

TL;DR

This paper introduces AQFormer, a novel audio-queried transformer for audio visual segmentation that explicitly models object-level correspondence and temporal interactions, significantly improving performance on AVS benchmarks.

Contribution

The paper proposes AQFormer, which uses audio-conditioned object queries and an audio-bridged temporal module to enhance sound object segmentation accuracy.

Findings

Achieves state-of-the-art results on AVS benchmarks.

Improves M_J and M_F metrics by over 7% on MS3 setting.

Demonstrates effectiveness of explicit object-level correspondence.

Abstract

Audio visual segmentation (AVS) aims to segment the sounding objects for each frame of a given video. To distinguish the sounding objects from silent ones, both audio-visual semantic correspondence and temporal interaction are required. The previous method applies multi-frame cross-modal attention to conduct pixel-level interactions between audio features and visual features of multiple frames simultaneously, which is both redundant and implicit. In this paper, we propose an Audio-Queried Transformer architecture, AQFormer, where we define a set of object queries conditioned on audio information and associate each of them to particular sounding objects. Explicit object-level semantic correspondence between audio and visual modalities is established by gathering object information from visual features with predefined audio queries. Besides, an Audio-Bridged Temporal Interaction module is proposed to exchange sounding object-relevant information among multiple frames with the bridge of audio features. Extensive experiments are conducted on two AVS benchmarks to show that our method achieves state-of-the-art performances, especially 7.1% M_J and 7.6% M_F gains on the MS3 setting.