StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion

TL;DR

StarGANv2-VC introduces a versatile, unsupervised, non-parallel voice conversion framework that produces natural, high-quality speech across various tasks, including cross-lingual and stylistic conversions, with real-time performance.

Contribution

It presents a novel GAN-based voice conversion model that generalizes well with limited data and can handle multiple conversion scenarios without parallel training data.

Findings

Outperforms previous VC models in naturalness and quality.

Generalizes to cross-lingual and stylistic voice conversions.

Operates in real-time with a faster-than-real-time vocoder.

Abstract

We present an unsupervised non-parallel many-to-many voice conversion (VC) method using a generative adversarial network (GAN) called StarGAN v2. Using a combination of adversarial source classifier loss and perceptual loss, our model significantly outperforms previous VC models. Although our model is trained only with 20 English speakers, it generalizes to a variety of voice conversion tasks, such as any-to-many, cross-lingual, and singing conversion. Using a style encoder, our framework can also convert plain reading speech into stylistic speech, such as emotional and falsetto speech. Subjective and objective evaluation experiments on a non-parallel many-to-many voice conversion task revealed that our model produces natural sounding voices, close to the sound quality of state-of-the-art text-to-speech (TTS) based voice conversion methods without the need for text labels. Moreover, our model is completely convolutional and with a faster-than-real-time vocoder such as Parallel WaveGAN can perform real-time voice conversion.