SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling

TL;DR

SSNAPS introduces a novel unsupervised audio-visual speech separation method using diffusion inverse sampling, outperforming supervised baselines in noisy environments and enabling applications like acoustic scene detection.

Contribution

The paper presents a new generative inverse sampling approach with diffusion priors for unsupervised speech and noise separation in complex audio-visual scenarios.

Findings

Outperforms supervised baselines in word error rate across various noisy conditions

Effectively separates off-screen speakers and ambient noise

Produces high-fidelity noise components suitable for scene detection

Abstract

This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise. Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources. To achieve this, we reformulate a recent inverse sampler to match our setting. We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in \ac{WER} across all conditions. We further extend our framework to handle off-screen speaker separation. Moreover, the high fidelity of the separated noise component makes it suitable for downstream acoustic scene detection. Demo page: https://ssnapsicml.github.io/ssnapsicml2026/