Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling

TL;DR

This paper presents a novel multi-factor disentanglement framework for expressive speech synthesis that effectively transfers style and emotion from reference speech to target speakers using a two-stage neural approach.

Contribution

It introduces a multi-factor decoupling method with multi-label binary vectors and mutual information minimization, along with semi-supervised training and an attention-based reference selection for improved expressive speech synthesis.

Findings

Effective disentanglement of speaker, style, and emotion factors.

High-quality style and emotion transfer in non-parallel data.

Robust synthesis across multiple speakers and expressions.

Abstract

This paper aims to synthesize the target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers. We address this challenging problem with a two-stage framework composed of a text-to-style-and-emotion (Text2SE) module and a style-and-emotion-to-wave (SE2Wave) module, bridging by neural bottleneck (BN) features. To further solve the multi-factor (speaker timbre, speaking style and emotion) decoupling problem, we adopt the multi-label binary vector (MBV) and mutual information (MI) minimization to respectively discretize the extracted embeddings and disentangle these highly entangled factors in both Text2SE and SE2Wave modules. Moreover, we introduce a semi-supervised training strategy to leverage data from multiple speakers, including emotion-labeled data, style-labeled data, and unlabeled data. To better transfer the fine-grained expression from references to the target speaker in non-parallel transfer, we introduce a reference-candidate pool and propose an attention-based reference selection approach. Extensive experiments demonstrate the good design of our model.