Libri2Vox Dataset: Target Speaker Extraction with Diverse Speaker Conditions and Synthetic Data

TL;DR

This paper introduces Libri2Vox, a diverse and realistic dataset for target speaker extraction, combined with synthetic data and curriculum learning to improve model robustness and performance in complex acoustic environments.

Contribution

The paper presents Libri2Vox, a novel dataset with real and synthetic speakers, and demonstrates how curriculum learning enhances TSE model performance using this data.

Findings

SpeakerBeam achieved a 1.39 dB SDR improvement on Libri2Talker.

Curriculum learning with the Conformer architecture further improved SDR by 0.78 dB.

Diverse real-world data and synthetic augmentation significantly enhance TSE robustness.

Abstract

Target speaker extraction (TSE) is essential in speech processing applications, particularly in scenarios with complex acoustic environments. Current TSE systems face challenges in limited data diversity and a lack of robustness in real-world conditions, primarily because they are trained on artificially mixed datasets with limited speaker variability and unrealistic noise profiles. To address these challenges, we propose Libri2Vox, a new dataset that combines clean target speech from the LibriTTS dataset with interference speech from the noisy VoxCeleb2 dataset, providing a large and diverse set of speakers under realistic noisy conditions. We also augment Libri2Vox with synthetic speakers generated using state-of-the-art speech generative models to enhance speaker diversity. Additionally, to further improve the effectiveness of incorporating synthetic data, curriculum learning is implemented to progressively train TSE models with increasing levels of difficulty. Extensive experiments across multiple TSE architectures reveal varying degrees of improvement, with SpeakerBeam demonstrating the most substantial gains: a 1.39 dB improvement in signal-to-distortion ratio (SDR) on the Libri2Talker test set compared to baseline training. Building upon these results, we further enhanced performance through our speaker similarity-based curriculum learning approach with the Conformer architecture, achieving an additional 0.78 dB improvement over conventional random sampling methods in which data samples are randomly selected from the entire dataset. These results demonstrate the complementary benefits of diverse real-world data, synthetic speaker augmentation, and structured training strategies in building robust TSE systems.