Improving Prosody for Unseen Texts in Speech Synthesis by Utilizing Linguistic Information and Noisy Data

TL;DR

This paper enhances speech synthesis prosody for unseen texts by combining a fine-tuned BERT front-end with a pre-trained FastSpeech2 model, leveraging noisy data and linguistic tasks to improve naturalness.

Contribution

It introduces a multi-task learning framework that fine-tunes BERT on linguistic tasks and pre-trains FastSpeech2 on noisy data to improve prosody in unseen text synthesis.

Findings

Improved prosody for complex sentences.

Enhanced naturalness in synthesized speech.

Effective use of noisy external data.

Abstract

Recent advancements in end-to-end speech synthesis have made it possible to generate highly natural speech. However, training these models typically requires a large amount of high-fidelity speech data, and for unseen texts, the prosody of synthesized speech is relatively unnatural. To address these issues, we propose to combine a fine-tuned BERT-based front-end with a pre-trained FastSpeech2-based acoustic model to improve prosody modeling. The pre-trained BERT is fine-tuned on the polyphone disambiguation task, the joint Chinese word segmentation (CWS) and part-of-speech (POS) tagging task, and the prosody structure prediction (PSP) task in a multi-task learning framework. FastSpeech 2 is pre-trained on large-scale external data that are noisy but easier to obtain. Experimental results show that both the fine-tuned BERT model and the pre-trained FastSpeech 2 can improve prosody, especially for those structurally complex sentences.