Analysis of Voice Conversion and Code-Switching Synthesis Using VQ-VAE

TL;DR

This paper analyzes the quality of multilingual speech synthesis using VQ-VAE, focusing on voice conversion and code-switching across four languages, revealing challenges like quality degradation with more switches and accent transfer effects.

Contribution

It provides a detailed analysis of multilingual VQ-VAE speech synthesis, highlighting limitations and potential directions for improvement in handling code-switching and voice conversion.

Findings

Speech quality decreases with more language switches

Accent transfer occurs during cross-language voice conversion

Assessment of accent transfer remains challenging

Abstract

This paper presents an analysis of speech synthesis quality achieved by simultaneously performing voice conversion and language code-switching using multilingual VQ-VAE speech synthesis in German, French, English and Italian. In this paper, we utilize VQ code indices representing phone information from VQ-VAE to perform code-switching and a VQ speaker code to perform voice conversion in a single system with a neural vocoder. Our analysis examines several aspects of code-switching including the number of language switches and the number of words involved in each switch. We found that speech synthesis quality degrades after increasing the number of language switches within an utterance and decreasing the number of words. We also found some evidence of accent transfer when performing voice conversion across languages as observed when a speaker's original language differs from the language of a synthetic target utterance. We present results from our listening tests and discuss the inherent difficulties of assessing accent transfer in speech synthesis. Our work highlights some of the limitations and strengths of using a semi-supervised end-to-end system like VQ-VAE for handling multilingual synthesis. Our work provides insight into why multilingual speech synthesis is challenging and we suggest some directions for expanding work in this area.