OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup

TL;DR

OmniSep is a unified framework for sound separation that uses omni-modal queries and a novel Query-Mixup training strategy to improve separation performance across multiple modalities and open-vocabulary scenarios.

Contribution

The paper introduces OmniSep, a novel multi-modal sound separation framework with Query-Mixup training and retrieval-augmented Query-Aug for open-vocabulary separation, advancing the state-of-the-art.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively handles multi-modal and open-vocabulary sound separation.

Demonstrates flexible query influence for sound retention or removal.

Abstract

The scaling up has brought tremendous success in the fields of vision and language in recent years. When it comes to audio, however, researchers encounter a major challenge in scaling up the training data, as most natural audio contains diverse interfering signals. To address this limitation, we introduce Omni-modal Sound Separation (OmniSep), a novel framework capable of isolating clean soundtracks based on omni-modal queries, encompassing both single-modal and multi-modal composed queries. Specifically, we introduce the Query-Mixup strategy, which blends query features from different modalities during training. This enables OmniSep to optimize multiple modalities concurrently, effectively bringing all modalities under a unified framework for sound separation. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds as desired. Finally, OmniSep employs a retrieval-augmented approach known as Query-Aug, which enables open-vocabulary sound separation. Experimental evaluations on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets demonstrate effectiveness of OmniSep, achieving state-of-the-art performance in text-, image-, and audio-queried sound separation tasks. For samples and further information, please visit the demo page at \url{https://omnisep.github.io/}.