Disentangled Speech Representation Learning for One-Shot Cross-lingual Voice Conversion Using $\beta$-VAE

TL;DR

This paper introduces an unsupervised $eta$-VAE-based method to disentangle speech into content and speaker representations, enabling effective one-shot cross-lingual voice conversion with demonstrated improvements in speech quality and speaker similarity.

Contribution

It presents a novel unsupervised learning approach using $eta$-VAE for speech disentanglement tailored for cross-lingual voice conversion, emphasizing architectural and dataset biases.

Findings

Effective disentanglement of speech content and speaker identity.

Improved voice conversion quality in cross-lingual, one-shot scenarios.

Both objective and subjective evaluations confirm method's success.

Abstract

We propose an unsupervised learning method to disentangle speech into content representation and speaker identity representation. We apply this method to the challenging one-shot cross-lingual voice conversion task to demonstrate the effectiveness of the disentanglement. Inspired by $\beta$-VAE, we introduce a learning objective that balances between the information captured by the content and speaker representations. In addition, the inductive biases from the architectural design and the training dataset further encourage the desired disentanglement. Both objective and subjective evaluations show the effectiveness of the proposed method in speech disentanglement and in one-shot cross-lingual voice conversion.