MulliVC: Multi-lingual Voice Conversion With Cycle Consistency

TL;DR

MulliVC is a novel multi-lingual voice conversion system that effectively disentangles speaker timbre from content and prosody without requiring multi-lingual paired data, using a cycle consistency approach.

Contribution

It introduces a three-step training process with cycle consistency to enable multi-lingual voice conversion without paired multi-lingual datasets.

Findings

Outperforms existing methods in monolingual and cross-lingual voice conversion

Effectively disentangles timbre from content and prosody

No need for multi-lingual paired data

Abstract

Voice conversion aims to modify the source speaker's voice to resemble the target speaker while preserving the original speech content. Despite notable advancements in voice conversion these days, multi-lingual voice conversion (including both monolingual and cross-lingual scenarios) has yet to be extensively studied. It faces two main challenges: 1) the considerable variability in prosody and articulation habits across languages; and 2) the rarity of paired multi-lingual datasets from the same speaker. In this paper, we propose MulliVC, a novel voice conversion system that only converts timbre and keeps original content and source language prosody without multi-lingual paired data. Specifically, each training step of MulliVC contains three substeps: In step one the model is trained with monolingual speech data; then, steps two and three take inspiration from back translation, construct a cyclical process to disentangle the timbre and other information (content, prosody, and other language-related information) in the absence of multi-lingual data from the same speaker. Both objective and subjective results indicate that MulliVC significantly surpasses other methods in both monolingual and cross-lingual contexts, demonstrating the system's efficacy and the viability of the three-step approach with cycle consistency. Audio samples can be found on our demo page (mullivc.github.io).