Exploring Timbre Disentanglement in Non-Autoregressive Cross-Lingual Text-to-Speech

TL;DR

This paper investigates how to effectively disentangle speaker and language features in non-autoregressive cross-lingual TTS models, introducing a phoneme length regulator and demonstrating improved naturalness and speaker similarity.

Contribution

We propose a phoneme length regulator and a FastPitch-based cross-lingual TTS model that enhances disentanglement of speaker and language representations.

Findings

Language-independent input representations improve disentanglement.

Increasing training speakers enhances model performance.

Explicit speech variance modeling benefits cross-lingual TTS.

Abstract

In this paper, we study the disentanglement of speaker and language representations in non-autoregressive cross-lingual TTS models from various aspects. We propose a phoneme length regulator that solves the length mismatch problem between IPA input sequence and monolingual alignment results. Using the phoneme length regulator, we present a FastPitch-based cross-lingual model with IPA symbols as input representations. Our experiments show that language-independent input representations (e.g. IPA symbols), an increasing number of training speakers, and explicit modeling of speech variance information all encourage non-autoregressive cross-lingual TTS model to disentangle speaker and language representations. The subjective evaluation shows that our proposed model can achieve decent naturalness and speaker similarity in cross-language voice cloning.