MF-Speech: Achieving Fine-Grained and Compositional Control in Speech Generation via Factor Disentanglement

TL;DR

MF-Speech introduces a novel framework that disentangles speech factors and enables fine-grained, compositional control in speech generation, significantly improving quality and controllability over previous methods.

Contribution

The paper proposes MF-Speech, a new framework with factor purification and hierarchical control, advancing speech generation by enabling precise, multi-factor manipulation and transferability of speech representations.

Findings

Outperforms state-of-the-art methods in multi-factor speech generation

Achieves lower word error rate (WER=4.67%) and higher subjective scores

Demonstrates strong transferability of learned speech factors

Abstract

Generating expressive and controllable human speech is one of the core goals of generative artificial intelligence, but its progress has long been constrained by two fundamental challenges: the deep entanglement of speech factors and the coarse granularity of existing control mechanisms. To overcome these challenges, we have proposed a novel framework called MF-Speech, which consists of two core components: MF-SpeechEncoder and MF-SpeechGenerator. MF-SpeechEncoder acts as a factor purifier, adopting a multi-objective optimization strategy to decompose the original speech signal into highly pure and independent representations of content, timbre, and emotion. Subsequently, MF-SpeechGenerator functions as a conductor, achieving precise, composable and fine-grained control over these factors through dynamic fusion and Hierarchical Style Adaptive Normalization (HSAN). Experiments demonstrate that in the highly challenging multi-factor compositional speech generation task, MF-Speech significantly outperforms current state-of-the-art methods, achieving a lower word error rate (WER=4.67%), superior style control (SECS=0.5685, Corr=0.68), and the highest subjective evaluation scores(nMOS=3.96, sMOS_emotion=3.86, sMOS_style=3.78). Furthermore, the learned discrete factors exhibit strong transferability, demonstrating their significant potential as a general-purpose speech representation.