Zero-Shot Long-Form Voice Cloning with Dynamic Convolution Attention

TL;DR

This paper introduces a zero-shot long-form voice cloning system that uses dynamic convolution attention to improve alignment and synthesis of very long speech utterances, maintaining naturalness and speaker similarity.

Contribution

It presents a novel attention mechanism combined with a modular system for zero-shot voice cloning capable of synthesizing long utterances with high quality.

Findings

Effective long-utterance synthesis with high intelligibility

Maintains naturalness and speaker similarity for short texts

Outperforms previous models in alignment and long-form speech quality

Abstract

With recent advancements in voice cloning, the performance of speech synthesis for a target speaker has been rendered similar to the human level. However, autoregressive voice cloning systems still suffer from text alignment failures, resulting in an inability to synthesize long sentences. In this work, we propose a variant of attention-based text-to-speech system that can reproduce a target voice from a few seconds of reference speech and generalize to very long utterances as well. The proposed system is based on three independently trained components: a speaker encoder, synthesizer and universal vocoder. Generalization to long utterances is realized using an energy-based attention mechanism known as Dynamic Convolution Attention, in combination with a set of modifications proposed for the synthesizer based on Tacotron 2. Moreover, effective zero-shot speaker adaptation is achieved by conditioning both the synthesizer and vocoder on a speaker encoder that has been pretrained on a large corpus of diverse data. We compare several implementations of voice cloning systems in terms of speech naturalness, speaker similarity, alignment consistency and ability to synthesize long utterances, and conclude that the proposed model can produce intelligible synthetic speech for extremely long utterances, while preserving a high extent of naturalness and similarity for short texts.