Improving curriculum learning for target speaker extraction with synthetic speakers

TL;DR

This paper enhances target speaker extraction by integrating synthetic speakers generated via voice conversion into curriculum learning, leading to improved model performance in complex speech environments.

Contribution

It introduces a k-nearest neighbor-based voice conversion method to generate diverse interference speakers for curriculum learning in TSE.

Findings

Synthetic speaker data improves TSE performance

Curriculum learning with synthetic data enhances model robustness

Significant accuracy gains in complex speech scenarios

Abstract

Target speaker extraction (TSE) aims to isolate individual speaker voices from complex speech environments. The effectiveness of TSE systems is often compromised when the speaker characteristics are similar to each other. Recent research has introduced curriculum learning (CL), in which TSE models are trained incrementally on speech samples of increasing complexity. In CL training, the model is first trained on samples with low speaker similarity between the target and interference speakers, and then on samples with high speaker similarity. To further improve CL, this paper uses a $k$-nearest neighbor-based voice conversion method to simulate and generate speech of diverse interference speakers, and then uses the generated data as part of the CL. Experiments demonstrate that training data based on synthetic speakers can effectively enhance the model's capabilities and significantly improve the performance of multiple TSE systems.