Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity

TL;DR

This paper introduces a novel sub-center speaker embedding approach that better captures intra-speaker variability, leading to more natural and expressive speech synthesis, especially in voice conversion tasks.

Contribution

It proposes a multi sub-center embedding method for speaker representations, enhancing intra-speaker variation modeling for speech generation.

Findings

Improved naturalness in synthesized speech.

Enhanced prosodic expressiveness.

Better intra-speaker variation capture.

Abstract

Modeling the rich prosodic variations inherent in human speech is essential for generating natural-sounding speech. While speaker embeddings are commonly used as conditioning inputs in personalized speech generation, they are typically optimized for speaker recognition, which encourages the loss of intra-speaker variation. This strategy makes them suboptimal for speech generation in terms of modeling the rich variations at the output speech distribution. In this work, we propose a novel speaker embedding network that employs multiple sub-centers per speaker class during training, instead of a single center as in conventional approaches. This sub-center modeling allows the embedding to capture a broader range of speaker-specific variations while maintaining speaker classification performance. We demonstrate the effectiveness of the proposed embeddings on a voice conversion task, showing improved naturalness and prosodic expressiveness in the synthesized speech.