Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching

TL;DR

This paper introduces R-VC, a novel zero-shot voice conversion model that effectively transfers rhythm and timbre with high quality and efficiency, even with limited data, by using advanced flow matching and content discretization techniques.

Contribution

R-VC is the first model to combine rhythm controllability with efficient zero-shot voice conversion using shortcut flow matching and content token discretization.

Findings

Achieves comparable speaker similarity to state-of-the-art methods.

Surpasses existing methods in speech naturalness and intelligibility.

Operates effectively with smaller datasets and fewer sampling steps.

Abstract

Zero-Shot Voice Conversion (VC) aims to transform the source speaker's timbre into an arbitrary unseen one while retaining speech content. Most prior work focuses on preserving the source's prosody, while fine-grained timbre information may leak through prosody, and transferring target prosody to synthesized speech is rarely studied. In light of this, we propose R-VC, a rhythm-controllable and efficient zero-shot voice conversion model. R-VC employs data perturbation techniques and discretize source speech into Hubert content tokens, eliminating much content-irrelevant information. By leveraging a Mask Generative Transformer for in-context duration modeling, our model adapts the linguistic content duration to the desired target speaking style, facilitating the transfer of the target speaker's rhythm. Furthermore, R-VC introduces a powerful Diffusion Transformer (DiT) with shortcut flow matching during training, conditioning the network not only on the current noise level but also on the desired step size, enabling high timbre similarity and quality speech generation in fewer sampling steps, even in just two, thus minimizing latency. Experimental results show that R-VC achieves comparable speaker similarity to state-of-the-art VC methods with a smaller dataset, and surpasses them in terms of speech naturalness, intelligibility and style transfer performance.