nnSpeech: Speaker-Guided Conditional Variational Autoencoder for Zero-shot Multi-speaker Text-to-Speech

TL;DR

nnSpeech introduces a zero-shot multi-speaker TTS model that synthesizes new speaker voices without fine-tuning, using only one reference utterance and a speaker-guided variational autoencoder to capture speaker and content features.

Contribution

The paper presents a novel speaker-guided conditional variational autoencoder for zero-shot multi-speaker TTS, eliminating the need for fine-tuning with minimal adaptation data.

Findings

Generates natural, similar speech with only one adaptation utterance.

Effective across multiple languages and datasets.

Outperforms existing methods in zero-shot speaker synthesis.

Abstract

Multi-speaker text-to-speech (TTS) using a few adaption data is a challenge in practical applications. To address that, we propose a zero-shot multi-speaker TTS, named nnSpeech, that could synthesis a new speaker voice without fine-tuning and using only one adaption utterance. Compared with using a speaker representation module to extract the characteristics of new speakers, our method bases on a speaker-guided conditional variational autoencoder and can generate a variable Z, which contains both speaker characteristics and content information. The latent variable Z distribution is approximated by another variable conditioned on reference mel-spectrogram and phoneme. Experiments on the English corpus, Mandarin corpus, and cross-dataset proves that our model could generate natural and similar speech with only one adaption speech.