Improve Bilingual TTS Using Dynamic Language and Phonology Embedding

TL;DR

This paper proposes a novel bilingual TTS system that uses dynamic language and phonology embeddings to improve pronunciation and intonation accuracy for English speech generated by a Chinese speaker.

Contribution

It introduces phonology embedding and an embedding strength modulator to effectively model language-specific pronunciation variations in bilingual TTS.

Findings

Significantly improved naturalness and standardness of English speech

Effective modeling of phonology differences enhances TTS performance

Dynamic embedding control benefits various bilingual TTS scenarios

Abstract

In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, the pronunciation and intonation of the second language are usually quite different due to the influence of the first language. Therefore, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingual Chinese speaker. We introduce phonology embedding to capture the English differences between different phonology. Embedding mask is applied to language embedding for distinguishing information between different languages and to phonology embedding for focusing on English expression. We specially design an embedding strength modulator to capture the dynamic strength of language and phonology. Experiments show that our approach can produce significantly more natural and standard spoken English speech of the monolingual Chinese speaker. From analysis, we find that suitable phonology control contributes to better performance in different scenarios.