REF-VC: Robust, Expressive and Fast Zero-Shot Voice Conversion with Diffusion Transformers

TL;DR

REF-VC is a novel zero-shot voice conversion system that combines noise robustness, expressiveness, and speed by using innovative strategies like random erasing, implicit alignment, and shortcut models, outperforming existing methods especially in noisy environments.

Contribution

The paper introduces REF-VC, a new voice conversion framework that enhances noise robustness and expressiveness while significantly reducing inference steps, with novel techniques like random erasing and implicit alignment.

Findings

Outperforms baselines like Seed-VC in noisy zero-shot scenarios

Maintains high quality on clean data comparable to Seed-VC

Enables singing voice conversion within one unified model

Abstract

In real-world voice conversion applications, environmental noise in source speech and user demands for expressive output pose critical challenges. Traditional ASR-based methods ensure noise robustness but suppress prosody richness, while SSL-based models improve expressiveness but suffer from timbre leakage and noise sensitivity. This paper proposes REF-VC, a noise-robust expressive voice conversion system. Key innovations include: (1) A random erasing strategy to mitigate the information redundancy inherent in SSL features, enhancing noise robustness and expressiveness; (2) Implicit alignment inspired by E2TTS to suppress non-essential feature reconstruction; (3) Integration of Shortcut Models to accelerate flow matching inference, significantly reducing to 4 steps. Experimental results demonstrate that REF-VC outperforms baselines such as Seed-VC in zero-shot scenarios on the noisy set, while also performing comparably to Seed-VC on the clean set. In addition, REF-VC can be compatible with singing voice conversion within one model.