Rapid Speaker Adaptation in Low Resource Text to Speech Systems using Synthetic Data and Transfer learning

TL;DR

This paper introduces a transfer learning and synthetic data approach for rapid speaker adaptation in low-resource TTS systems, enabling high-quality Hindi speech synthesis with minimal target data.

Contribution

It proposes a novel three-step transfer learning method combining high-resource language data and synthetic data for low-resource TTS adaptation.

Findings

Effective speaker adaptation with only 3 hours of data

Synthetic data improves TTS quality in low-resource settings

Transfer learning from high-resource language accelerates model training

Abstract

Text-to-speech (TTS) systems are being built using end-to-end deep learning approaches. However, these systems require huge amounts of training data. We present our approach to built production quality TTS and perform speaker adaptation in extremely low resource settings. We propose a transfer learning approach using high-resource language data and synthetically generated data. We transfer the learnings from the out-domain high-resource English language. Further, we make use of out-of-the-box single-speaker TTS in the target language to generate in-domain synthetic data. We employ a three-step approach to train a high-quality single-speaker TTS system in a low-resource Indian language Hindi. We use a Tacotron2 like setup with a spectrogram prediction network and a waveglow vocoder. The Tacotron2 acoustic model is trained on English data, followed by synthetic Hindi data from the existing TTS system. Finally, the decoder of this model is fine-tuned on only 3 hours of target Hindi speaker data to enable rapid speaker adaptation. We show the importance of this dual pre-training and decoder-only fine-tuning using subjective MOS evaluation. Using transfer learning from high-resource language and synthetic corpus we present a low-cost solution to train a custom TTS model.