ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly Disentangled Self-supervised Speech Representations

TL;DR

This paper introduces ACE-VC, a zero-shot voice conversion method that uses self-supervised speech representations and disentangles content and speaker features for controllable, high-quality voice conversion with minimal data.

Contribution

It presents a novel multi-task, self-supervised framework with a Siamese network training strategy for disentangling speech features, enabling zero-shot, any-to-any voice conversion.

Findings

Achieves state-of-the-art speaker similarity, intelligibility, and naturalness metrics.

Performs effective voice swapping with only 10 seconds of target data.

Attains low speaker verification EER for both seen and unseen speakers.

Abstract

In this work, we propose a zero-shot voice conversion method using speech representations trained with self-supervised learning. First, we develop a multi-task model to decompose a speech utterance into features such as linguistic content, speaker characteristics, and speaking style. To disentangle content and speaker representations, we propose a training strategy based on Siamese networks that encourages similarity between the content representations of the original and pitch-shifted audio. Next, we develop a synthesis model with pitch and duration predictors that can effectively reconstruct the speech signal from its decomposed representation. Our framework allows controllable and speaker-adaptive synthesis to perform zero-shot any-to-any voice conversion achieving state-of-the-art results on metrics evaluating speaker similarity, intelligibility, and naturalness. Using just 10 seconds of data for a target speaker, our framework can perform voice swapping and achieves a speaker verification EER of 5.5% for seen speakers and 8.4% for unseen speakers.