Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model

TL;DR

This paper presents PFlow-VC, a novel voice conversion model that uses discrete pitch tokens and target speaker prompts to enhance expressiveness, style transfer, and timbre similarity in speech synthesis.

Contribution

It introduces a simple, efficient approach combining self-supervised pitch discretization and flow matching for improved expressive voice conversion.

Findings

Outperforms previous models in timbre and style transfer on LibriTTS and ESD datasets.

Effectively models in-context pitch for more natural voice conversion.

Enhances timbre similarity by integrating global and time-varying timbre embeddings.

Abstract

This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page https://speechai-demo.github.io/PFlow-VC/.