Multi-modal Adversarial Training for Zero-Shot Voice Cloning

TL;DR

This paper introduces a novel adversarial training method using a Transformer-based discriminator to enhance zero-shot voice cloning in TTS systems, significantly improving speech naturalness and speaker similarity.

Contribution

It proposes a new adversarial training approach with a Transformer discriminator for zero-shot voice cloning, improving upon existing GAN-based methods.

Findings

Enhanced speech quality and naturalness

Improved speaker similarity in zero-shot cloning

Effective training on large multi-speaker datasets

Abstract

A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are close to the average characteristics of a dataset, failing to model the variations that make human speech sound natural. This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance in speaking styles. We build off of recent works which have used Generative Advsarial Networks (GAN) by proposing a Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech features. The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a TTS model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model and training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model achieves improvements over the baseline in terms of speech quality and speaker similarity. Audio examples from our system are available online.