An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios

TL;DR

This paper investigates how self-supervised multilingual models can be adapted for low-resource language TTS, analyzing factors affecting performance and revealing insights into optimal fine-tuning strategies.

Contribution

It provides an empirical analysis of language adaptation in SSL-based TTS systems, highlighting the impact of phonetic similarity, dataset size, and data pairing on adaptation success.

Findings

Phonetic similarity influences adaptation performance.

Dataset size and speaker diversity affect TTS adaptation.

Paired data is not always better than audio-only data for fine-tuning.

Abstract

Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS.