Unsupervised Single-Channel Audio Separation with Diffusion Source Priors

TL;DR

This paper introduces an unsupervised method for single-channel audio separation using diffusion source priors, leveraging a novel inverse problem solver and a time-frequency attention network to improve separation quality without relying on paired training data.

Contribution

The work presents a new unsupervised approach for audio separation that uses diffusion priors and an advanced inverse solver, eliminating the need for paired training data and enhancing separation performance.

Findings

Achieved high-quality separation across speech and sound event tasks.

Significant performance improvements over existing unsupervised methods.

Effective initialization with augmented mixtures enhances results.

Abstract

Single-channel audio separation aims to separate individual sources from a single-channel mixture. Most existing methods rely on supervised learning with synthetically generated paired data. However, obtaining high-quality paired data in real-world scenarios is often difficult. This data scarcity can degrade model performance under unseen conditions and limit generalization ability. To this end, in this work, we approach this problem from an unsupervised perspective, framing it as a probabilistic inverse problem. Our method requires only diffusion priors trained on individual sources. Separation is then achieved by iteratively guiding an initial state toward the solution through reconstruction guidance. Importantly, we introduce an advanced inverse problem solver specifically designed for separation, which mitigates gradient conflicts caused by interference between the diffusion prior and reconstruction guidance during inverse denoising. This design ensures high-quality and balanced separation performance across individual sources. Additionally, we find that initializing the denoising process with an augmented mixture instead of pure Gaussian noise provides an informative starting point that significantly improves the final performance. To further enhance audio prior modeling, we design a novel time-frequency attention-based network architecture that demonstrates strong audio modeling capability. Collectively, these improvements lead to significant performance gains, as validated across speech-sound event, sound event, and speech separation tasks.