Text Enhancement for Paragraph Processing in End-to-End Code-switching TTS

TL;DR

This paper introduces text enhancement and cross-lingual embedding techniques to improve naturalness, consistency, and prosody stability in end-to-end code-switching TTS systems across multiple language pairs.

Contribution

It proposes novel methods for text enhancement and cross-lingual embeddings that significantly improve code-switching TTS quality and can be extended to various languages.

Findings

Enhanced naturalness and consistency in code-switching speech

Improved prosody stability in paragraph synthesis

Effective across multiple language pairs including Mandarin-English, Shanghaiese, and Cantonese

Abstract

Current end-to-end code-switching Text-to-Speech (TTS) can already generate high quality two languages speech in the same utterance with single speaker bilingual corpora. When the speakers of the bilingual corpora are different, the naturalness and consistency of the code-switching TTS will be poor. The cross-lingual embedding layers structure we proposed makes similar syllables in different languages relevant, thus improving the naturalness and consistency of generated speech. In the end-to-end code-switching TTS, there exists problem of prosody instability when synthesizing paragraph text. The text enhancement method we proposed makes the input contain prosodic information and sentence-level context information, thus improving the prosody stability of paragraph text. Experimental results demonstrate the effectiveness of the proposed methods in the naturalness, consistency, and prosody stability. In addition to Mandarin and English, we also apply these methods to Shanghaiese and Cantonese corpora, proving that the methods we proposed can be extended to other languages to build end-to-end code-switching TTS system.