AdaSpeech 2: Adaptive Text to Speech with Untranscribed Data

TL;DR

AdaSpeech 2 introduces an adaptive TTS system that effectively adapts to new speakers using only untranscribed speech data, leveraging a mel-spectrogram encoder for speech reconstruction and fine-tuning the decoder.

Contribution

The paper presents a novel untranscribed data adaptation method for TTS that is pluggable and achieves comparable or better voice quality than transcribed data methods.

Findings

Achieves on-par voice quality with transcribed adaptation.

Outperforms previous untranscribed adaptation methods.

Can be applied to existing TTS models without re-training.

Abstract

Text to speech (TTS) is widely used to synthesize personal voice for a target speaker, where a well-trained source TTS model is fine-tuned with few paired adaptation data (speech and its transcripts) on this target speaker. However, in many scenarios, only untranscribed speech data is available for adaptation, which brings challenges to the previous TTS adaptation pipelines (e.g., AdaSpeech). In this paper, we develop AdaSpeech 2, an adaptive TTS system that only leverages untranscribed speech data for adaptation. Specifically, we introduce a mel-spectrogram encoder to a well-trained TTS model to conduct speech reconstruction, and at the same time constrain the output sequence of the mel-spectrogram encoder to be close to that of the original phoneme encoder. In adaptation, we use untranscribed speech data for speech reconstruction and only fine-tune the TTS decoder. AdaSpeech 2 has two advantages: 1) Pluggable: our system can be easily applied to existing trained TTS models without re-training. 2) Effective: our system achieves on-par voice quality with the transcribed TTS adaptation (e.g., AdaSpeech) with the same amount of untranscribed data, and achieves better voice quality than previous untranscribed adaptation methods. Synthesized speech samples can be found at https://speechresearch.github.io/adaspeech2/.