PromptSep: Generative Audio Separation via Multimodal Prompting

TL;DR

PromptSep advances audio source separation by integrating multimodal prompts, including vocal imitation, to enable more intuitive and versatile sound extraction and removal, outperforming previous models in various benchmarks.

Contribution

The paper introduces PromptSep, a novel framework that combines language and vocal imitation prompts with a diffusion model for improved, flexible audio separation tasks.

Findings

State-of-the-art sound removal performance

Effective vocal imitation-guided separation

Competitive language-queried separation results

Abstract

Recent breakthroughs in language-queried audio source separation (LASS) have shown that generative models can achieve higher separation audio quality than traditional masking-based approaches. However, two key limitations restrict their practical use: (1) users often require operations beyond separation, such as sound removal; and (2) relying solely on text prompts can be unintuitive for specifying sound sources. In this paper, we propose PromptSep to extend LASS into a broader framework for general-purpose sound separation. PromptSep leverages a conditional diffusion model enhanced with elaborated data simulation to enable both audio extraction and sound removal. To move beyond text-only queries, we incorporate vocal imitation as an additional and more intuitive conditioning modality for our model, by incorporating Sketch2Sound as a data augmentation strategy. Both objective and subjective evaluations on multiple benchmarks demonstrate that PromptSep achieves state-of-the-art performance in sound removal and vocal-imitation-guided source separation, while maintaining competitive results on language-queried source separation.