Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic and Textual Contexts

TL;DR

This paper introduces a multi-modal context-aware Japanese audiobook TTS system that enhances prosody by integrating acoustic and textual contexts, outperforming previous methods through extensive evaluations.

Contribution

It proposes a novel multimodal context encoding approach for TTS that incorporates both acoustic and textual information to improve prosody, with new insights on context modality choices.

Findings

Significant improvement over previous TTS methods in prosody quality.

Effective use of multimodal context encoding for speech synthesis.

Insights into modality, lateral information, and context length choices.

Abstract

We present a multi-speaker Japanese audiobook text-to-speech (TTS) system that leverages multimodal context information of preceding acoustic context and bilateral textual context to improve the prosody of synthetic speech. Previous work either uses unilateral or single-modality context, which does not fully represent the context information. The proposed method uses an acoustic context encoder and a textual context encoder to aggregate context information and feeds it to the TTS model, which enables the model to predict context-dependent prosody. We conducted comprehensive objective and subjective evaluations on a multi-speaker Japanese audiobook dataset. Experimental results demonstrate that the proposed method significantly outperforms two previous works. Additionally, we present insights about the different choices of context - modalities, lateral information and length - for audiobook TTS that have never been discussed in the literature before.