Target Speech Extraction with Conditional Diffusion Model

TL;DR

This paper introduces a novel target speech extraction method using a conditional diffusion model, which outperforms existing discriminative systems and incorporates ensemble inference to enhance accuracy in multi-talker scenarios.

Contribution

The paper presents a new diffusion model-based approach for target speech extraction conditioned on speaker clues, with ensemble inference to improve performance.

Findings

Outperforms discriminative TSE systems on Libri2mix

Ensemble inference reduces extraction errors

Diffusion models generate more natural speech signals

Abstract

Diffusion model-based speech enhancement has received increased attention since it can generate very natural enhanced signals and generalizes well to unseen conditions. Diffusion models have been explored for several sub-tasks of speech enhancement, such as speech denoising, dereverberation, and source separation. In this paper, we investigate their use for target speech extraction (TSE), which consists of estimating the clean speech signal of a target speaker in a mixture of multi-talkers. TSE is realized by conditioning the extraction process on a clue identifying the target speaker. We show we can realize TSE using a conditional diffusion model conditioned on the clue. Besides, we introduce ensemble inference to reduce potential extraction errors caused by the diffusion process. In experiments on Libri2mix corpus, we show that the proposed diffusion model-based TSE combined with ensemble inference outperforms a comparable TSE system trained discriminatively.