Unify and Conquer: How Phonetic Feature Representation Affects Polyglot Text-To-Speech (TTS)

TL;DR

This paper compares unified and separate phonetic feature representations in multilingual neural TTS systems, finding that unified representations generally outperform separate ones in naturalness and accent, especially with larger embedding sizes.

Contribution

It provides a comprehensive comparison of unified versus separate phonetic representations in multilingual NTTS, demonstrating the advantages of unified approaches for cross-lingual synthesis.

Findings

Unified representations improve cross-lingual naturalness and accent.

Separate representations have more tokens and may limit model capacity.

The benefit of unified representations emerges at larger embedding sizes.

Abstract

An essential design decision for multilingual Neural Text-To-Speech (NTTS) systems is how to represent input linguistic features within the model. Looking at the wide variety of approaches in the literature, two main paradigms emerge, unified and separate representations. The former uses a shared set of phonetic tokens across languages, whereas the latter uses unique phonetic tokens for each language. In this paper, we conduct a comprehensive study comparing multilingual NTTS systems models trained with both representations. Our results reveal that the unified approach consistently achieves better cross-lingual synthesis with respect to both naturalness and accent. Separate representations tend to have an order of magnitude more tokens than unified ones, which may affect model capacity. For this reason, we carry out an ablation study to understand the interaction of the representation type with the size of the token embedding. We find that the difference between the two paradigms only emerges above a certain threshold embedding size. This study provides strong evidence that unified representations should be the preferred paradigm when building multilingual NTTS systems.