Conditional Diffusion Model for Target Speaker Extraction

TL;DR

This paper introduces DiffSpEx, a novel generative diffusion-based method for extracting target speakers from mixed audio, leveraging stochastic differential equations and speaker embeddings, with promising results on standard datasets.

Contribution

The paper presents DiffSpEx, a new diffusion model for target speaker extraction that incorporates speaker embeddings and achieves improved performance and personalization capabilities.

Findings

Achieves 12.9 dB SI-SDR on WSJ0-2mix

Demonstrates effective speaker personalization through fine-tuning

Scores 3.56 on NISQA quality metric

Abstract

We propose DiffSpEx, a generative target speaker extraction method based on score-based generative modelling through stochastic differential equations. DiffSpEx deploys a continuous-time stochastic diffusion process in the complex short-time Fourier transform domain, starting from the target speaker source and converging to a Gaussian distribution centred on the mixture of sources. For the reverse-time process, a parametrised score function is conditioned on a target speaker embedding to extract the target speaker from the mixture of sources. We utilise ECAPA-TDNN target speaker embeddings and condition the score function alternately on the SDE time embedding and the target speaker embedding. The potential of DiffSpEx is demonstrated with the WSJ0-2mix dataset, achieving an SI-SDR of 12.9 dB and a NISQA score of 3.56. Moreover, we show that fine-tuning a pre-trained DiffSpEx model to a specific speaker further improves performance, enabling personalisation in target speaker extraction.