Referee: Towards reference-free cross-speaker style transfer with low-quality data for expressive speech synthesis

TL;DR

Referee introduces a reference-free, robust cross-speaker style transfer method for expressive speech synthesis that leverages low-quality data and combines text-to-style and style-to-wave models for high-fidelity output.

Contribution

It proposes a novel cascade approach using a text-to-style model with a pretrain-refinement method and a style-to-wave model, enabling style transfer without high-quality reference data.

Findings

Referee outperforms GST-based baseline in style transfer quality.

Effective learning of speaking styles from low-quality data.

High-fidelity speech synthesis achieved with the proposed method.

Abstract

Cross-speaker style transfer (CSST) in text-to-speech (TTS) synthesis aims at transferring a speaking style to the synthesised speech in a target speaker's voice. Most previous CSST approaches rely on expensive high-quality data carrying desired speaking style during training and require a reference utterance to obtain speaking style descriptors as conditioning on the generation of a new sentence. This work presents Referee, a robust reference-free CSST approach for expressive TTS, which fully leverages low-quality data to learn speaking styles from text. Referee is built by cascading a text-to-style (T2S) model with a style-to-wave (S2W) model. Phonetic PosteriorGram (PPG), phoneme-level pitch and energy contours are adopted as fine-grained speaking style descriptors, which are predicted from text using the T2S model. A novel pretrain-refinement method is adopted to learn a robust T2S model by only using readily accessible low-quality data. The S2W model is trained with high-quality target data, which is adopted to effectively aggregate style descriptors and generate high-fidelity speech in the target speaker's voice. Experimental results are presented, showing that Referee outperforms a global-style-token (GST)-based baseline approach in CSST.