Towards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

TL;DR

This paper introduces a multi-scale speaking style modeling approach for Mandarin speech synthesis, capturing hierarchical style information to enhance naturalness and expressiveness of synthetic speech.

Contribution

It proposes a novel multi-scale style extractor and predictor that utilize hierarchical context, addressing the neglect of multi-scale style in previous models.

Findings

Significant improvement in speech naturalness and expressiveness

Effective multi-scale style embedding extraction from ground-truth speech

Hierarchical context information enhances style prediction accuracy

Abstract

Previous works on expressive speech synthesis focus on modelling the mono-scale style embedding from the current sentence or context, but the multi-scale nature of speaking style in human speech is neglected. In this paper, we propose a multi-scale speaking style modelling method to capture and predict multi-scale speaking style for improving the naturalness and expressiveness of synthetic speech. A multi-scale extractor is proposed to extract speaking style embeddings at three different levels from the ground-truth speech, and explicitly guide the training of a multi-scale style predictor based on hierarchical context information. Both objective and subjective evaluations on a Mandarin audiobooks dataset demonstrate that our proposed method can significantly improve the naturalness and expressiveness of the synthesized speech.