Target Speaker Extraction through Comparing Noisy Positive and Negative Audio Enrollments

TL;DR

This paper introduces a novel method for target speaker extraction that leverages noisy enrollment segments by comparing positive and negative speech segments, achieving state-of-the-art results without requiring clean enrollment data.

Contribution

The work proposes a new enrollment strategy using noisy positive and negative segments and a two-stage training process, advancing target speaker extraction in realistic noisy conditions.

Findings

Achieves over 2.1 dB higher SI-SNRi than previous methods.

Reduces training convergence time by 60%.

State-of-the-art performance in noisy enrollment scenarios.

Abstract

Target speaker extraction focuses on isolating a specific speaker's voice from an audio mixture containing multiple speakers. To provide information about the target speaker's identity, prior works have utilized clean audio samples as conditioning inputs. However, such clean audio examples are not always readily available. For instance, obtaining a clean recording of a stranger's voice at a cocktail party without leaving the noisy environment is generally infeasible. Limited prior research has explored extracting the target speaker's characteristics from noisy enrollments, which may contain overlapping speech from interfering speakers. In this work, we explore a novel enrollment strategy that encodes target speaker information from the noisy enrollment by comparing segments where the target speaker is talking (Positive Enrollments) with segments where the target speaker is silent (Negative Enrollments). Experiments show the effectiveness of our model architecture, which achieves over 2.1 dB higher SI-SNRi compared to prior works in extracting the monaural speech from the mixture of two speakers. Additionally, the proposed two-stage training strategy accelerates convergence, reducing the number of optimization steps required to reach 3 dB SNR by 60%. Overall, our method achieves state-of-the-art performance in the monaural target speaker extraction conditioned on noisy enrollments. Our implementation is available at https://github.com/xu-shitong/TSE-through-Positive-Negative-Enroll .