MSStyleTTS: Multi-Scale Style Modeling with Hierarchical Context Information for Expressive Speech Synthesis

TL;DR

MSStyleTTS introduces a hierarchical multi-scale style modeling approach that leverages broader context for more natural and expressive speech synthesis, outperforming existing methods on audiobook datasets.

Contribution

The paper presents a novel multi-scale style modeling framework that incorporates hierarchical context information for expressive speech synthesis, which was not addressed in prior works.

Findings

Significant improvement over baseline methods in naturalness and expressiveness.

Effective modeling of multi-scale style embeddings from broader context.

Analysis of hierarchical context and style representations enhances understanding of speech expressiveness.

Abstract

Expressive speech synthesis is crucial for many human-computer interaction scenarios, such as audiobooks, podcasts, and voice assistants. Previous works focus on predicting the style embeddings at one single scale from the information within the current sentence. Whereas, context information in neighboring sentences and multi-scale nature of style in human speech are neglected, making it challenging to convert multi-sentence text into natural and expressive speech. In this paper, we propose MSStyleTTS, a style modeling method for expressive speech synthesis, to capture and predict styles at different levels from a wider range of context rather than a sentence. Two sub-modules, including multi-scale style extractor and multi-scale style predictor, are trained together with a FastSpeech 2 based acoustic model. The predictor is designed to explore the hierarchical context information by considering structural relationships in context and predict style embeddings at global-level, sentence-level and subword-level. The extractor extracts multi-scale style embedding from the ground-truth speech and explicitly guides the style prediction. Evaluations on both in-domain and out-of-domain audiobook datasets demonstrate that the proposed method significantly outperforms the three baselines. In addition, we conduct the analysis of the context information and multi-scale style representations that have never been discussed before.