PseudoVC: Improving One-shot Voice Conversion with Pseudo Paired Data

TL;DR

PseudoVC introduces a novel training approach for one-shot voice conversion that uses pseudo paired data through information perturbation and speaker sampling, significantly improving conversion quality.

Contribution

The paper presents PseudoVC, a new training method that addresses input mismatches in one-shot VC using pseudo conversion and speaker sampling techniques.

Findings

Pseudo Conversion outperforms previous information perturbation methods.

PseudoVC surpasses existing VC models in quality.

Experimental results validate the effectiveness of the proposed approach.

Abstract

As parallel training data is scarce for one-shot voice conversion (VC) tasks, waveform reconstruction is typically performed by various VC systems. A typical one-shot VC system comprises a content encoder and a speaker encoder. However, two types of mismatches arise: one for the inputs to the content encoder during training and inference, and another for the inputs to the speaker encoder. To address these mismatches, we propose a novel VC training method called \textit{PseudoVC} in this paper. First, we introduce an innovative information perturbation approach named \textit{Pseudo Conversion} to tackle the first mismatch problem. This approach leverages pretrained VC models to convert the source utterance into a perturbed utterance, which is fed into the content encoder during training. Second, we propose an approach termed \textit{Speaker Sampling} to resolve the second mismatch problem, which will substitute the input to the speaker encoder by another utterance from the same speaker during training. Experimental results demonstrate that our proposed \textit{Pseudo Conversion} outperforms previous information perturbation methods, and the overall \textit{PseudoVC} method surpasses publicly available VC models. Audio examples are available.