ZeroSep: Separate Anything in Audio with Zero Training

TL;DR

ZeroSep demonstrates that pre-trained text-guided audio diffusion models can perform zero-shot source separation without any task-specific training, effectively handling open-set scenarios and outperforming supervised methods.

Contribution

The paper introduces ZeroSep, a novel approach that repurposes pre-trained diffusion models for audio source separation in a zero-shot setting, eliminating the need for labeled data.

Findings

ZeroSep achieves competitive separation performance without training.

Supports open-set scenarios through rich textual priors.

Outperforms supervised methods on multiple benchmarks.

Abstract

Audio source separation is fundamental for machines to understand complex acoustic environments and underpins numerous audio applications. Current supervised deep learning approaches, while powerful, are limited by the need for extensive, task-specific labeled data and struggle to generalize to the immense variability and open-set nature of real-world acoustic scenes. Inspired by the success of generative foundation models, we investigate whether pre-trained text-guided audio diffusion models can overcome these limitations. We make a surprising discovery: zero-shot source separation can be achieved purely through a pre-trained text-guided audio diffusion model under the right configuration. Our method, named ZeroSep, works by inverting the mixed audio into the diffusion model's latent space and then using text conditioning to guide the denoising process to recover individual sources. Without any task-specific training or fine-tuning, ZeroSep repurposes the generative diffusion model for a discriminative separation task and inherently supports open-set scenarios through its rich textual priors. ZeroSep is compatible with a variety of pre-trained text-guided audio diffusion backbones and delivers strong separation performance on multiple separation benchmarks, surpassing even supervised methods.