GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech

TL;DR

GenerSpeech introduces a novel TTS model that effectively performs high-fidelity zero-shot style transfer for out-of-domain speech, handling diverse styles and improving robustness over existing methods.

Contribution

The paper presents a new TTS framework with multi-level style adaptation and a generalizable content adaptor, enabling superior zero-shot style transfer for out-of-domain speech.

Findings

Outperforms state-of-the-art models in audio quality and style similarity.

Demonstrates robustness in few-shot adaptive style transfer.

Effectively models diverse style conditions including speaker, emotion, and prosody.

Abstract

Style transfer for out-of-domain (OOD) speech synthesis aims to generate speech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from an acoustic reference, while facing the following challenges: 1) The highly dynamic style features in expressive voice are difficult to model and transfer; and 2) the TTS models should be robust enough to handle diverse OOD conditions that differ from the source data. This paper proposes GenerSpeech, a text-to-speech model towards high-fidelity zero-shot style transfer of OOD custom voice. GenerSpeech decomposes the speech variation into the style-agnostic and style-specific parts by introducing two components: 1) a multi-level style adaptor to efficiently model a large range of style conditions, including global speaker and emotion characteristics, and the local (utterance, phoneme, and word-level) fine-grained prosodic representations; and 2) a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation and thus improve model generalization. Our evaluations on zero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-art models in terms of audio quality and style similarity. The extension studies to adaptive style transfer further show that GenerSpeech performs robustly in the few-shot data setting. Audio samples are available at https://GenerSpeech.github.io/