GenVC: Self-Supervised Zero-Shot Voice Conversion

TL;DR

GenVC is a self-supervised zero-shot voice conversion framework that disentangles speaker identity from content, achieving high speaker similarity and privacy protection without external supervision, using speech tokenizers and Transformer models.

Contribution

It introduces a novel self-supervised approach for voice conversion that removes the need for external speaker encoders, utilizing speech tokenizers and autoregressive Transformers.

Findings

Higher speaker similarity than existing methods

Maintains naturalness comparable to top zero-shot approaches

Enhances privacy protection and voice anonymization

Abstract

Most current zero-shot voice conversion methods rely on externally supervised components, particularly speaker encoders, for training. To explore alternatives that eliminate this dependency, this paper introduces GenVC, a novel framework that disentangles speaker identity and linguistic content from speech signals in a self-supervised manner. GenVC leverages speech tokenizers and an autoregressive, Transformer-based language model as its backbone for speech generation. This design supports large-scale training while enhancing both source speaker privacy protection and target speaker cloning fidelity. Experimental results demonstrate that GenVC achieves notably higher speaker similarity, with naturalness on par with leading zero-shot approaches. Moreover, due to its autoregressive formulation, GenVC introduces flexibility in temporal alignment, reducing the preservation of source prosody and speaker-specific traits, and making it highly effective for voice anonymization.