Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling

TL;DR

This paper explores the use of a phoneme-level RVQ-VAE model for speech prosody modeling, demonstrating high disentanglement and interpretability of prosodic features like pitch and energy.

Contribution

It introduces a novel phoneme-level RVQ-VAE approach that effectively disentangles prosodic features from phonetic and speaker information.

Findings

High degree of prosody disentanglement achieved

Latent space components correspond to pitch and energy

Robust and transferable prosodic representations

Abstract

Most of the prevalent approaches in speech prosody modeling rely on learning global style representations in a continuous latent space which encode and transfer the attributes of reference speech. However, recent work on neural codecs which are based on Residual Vector Quantization (RVQ) already shows great potential offering distinct advantages. We investigate the prosody modeling capabilities of the discrete space of such an RVQ-VAE model, modifying it to operate on the phoneme-level. We condition both the encoder and decoder of the model on linguistic representations and apply a global speaker embedding in order to factor out both phonetic and speaker information. We conduct an extensive set of investigations based on subjective experiments and objective measures to show that the phoneme-level discrete latent representations obtained this way achieves a high degree of disentanglement, capturing fine-grained prosodic information that is robust and transferable. The latent space turns out to have interpretable structure with its principal components corresponding to pitch and energy.