iQuery: Instruments as Queries for Audio-Visual Sound Separation

TL;DR

This paper introduces iQuery, a novel audio-visual sound separation method using instrument-as-query prompts, which enhances generalization to new instruments and improves separation performance across benchmarks.

Contribution

The paper proposes a flexible query-based framework for audio-visual separation that disentangles multi-modal features and enables generalization to new instruments without full network finetuning.

Findings

Improves separation performance on three benchmarks.

Enables generalization to unseen instruments with minimal fine-tuning.

Uses cross-modal attention to reduce source interference.

Abstract

Current audio-visual separation methods share a standard architecture design where an audio encoder-decoder network is fused with visual encoding features at the encoder bottleneck. This design confounds the learning of multi-modal feature encoding with robust sound decoding for audio separation. To generalize to a new instrument: one must finetune the entire visual and audio network for all musical instruments. We re-formulate visual-sound separation task and propose Instrument as Query (iQuery) with a flexible query expansion mechanism. Our approach ensures cross-modal consistency and cross-instrument disentanglement. We utilize "visually named" queries to initiate the learning of audio queries and use cross-modal attention to remove potential sound source interference at the estimated waveforms. To generalize to a new instrument or event class, drawing inspiration from the text-prompt design, we insert an additional query as an audio prompt while freezing the attention mechanism. Experimental results on three benchmarks demonstrate that our iQuery improves audio-visual sound source separation performance.