Adaptive Data Augmentation with NaturalSpeech3 for Far-field Speaker Verification

TL;DR

This paper introduces an adaptive data augmentation method using NaturalSpeech3 to convert near-field speech into realistic far-field speech with ambient noise, improving far-field speaker verification performance.

Contribution

The novel approach leverages NaturalSpeech3's disentangled embeddings to generate pseudo far-field speech that enhances training data for speaker verification systems.

Findings

Significant performance improvement over traditional augmentation methods.

Effective preservation of speaker identity in augmented speech.

Enhanced robustness of speaker verification in far-field scenarios.

Abstract

The scarcity of speaker-annotated far-field speech presents a significant challenge in developing high-performance far-field speaker verification (SV) systems. While data augmentation using large-scale near-field speech has been a common strategy to address this limitation, the mismatch in acoustic environments between near-field and far-field speech significantly hinders the improvement of far-field SV effectiveness. In this paper, we propose an adaptive speech augmentation approach leveraging NaturalSpeech3, a pre-trained foundation text-to-speech (TTS) model, to convert near-field speech into far-field speech by incorporating far-field acoustic ambient noise for data augmentation. Specifically, we utilize FACodec from NaturalSpeech3 to decompose the speech waveform into distinct embedding subspaces-content, prosody, speaker, and residual (acoustic details) embeddings-and reconstruct the speech waveform from these disentangled representations. In our method, the prosody, content, and residual embeddings of far-field speech are combined with speaker embeddings from near-field speech to generate augmented pseudo far-field speech that maintains the speaker identity from the out-domain near-field speech while preserving the acoustic environment of the in-domain far-field speech. This approach not only serves as an effective strategy for augmenting training data for far-field speaker verification but also extends to cross-data augmentation for enrollment and test speech in evaluation trials.Experimental results on FFSVC demonstrate that the adaptive data augmentation method significantly outperforms traditional approaches, such as random noise addition and reverberation, as well as other competitive data augmentation strategies.