Emotion-Coherent Speech Data Augmentation and Self-Supervised Contrastive Style Training for Enhancing Kids's Story Speech Synthesis

TL;DR

This paper introduces a novel data augmentation and self-supervised contrastive training approach to improve expressive kids's story speech synthesis, resulting in more natural and style-consistent multi-sentence speech generation.

Contribution

It presents a new method combining emotionally congruent audio merging and contrastive style training to enhance expressive speech synthesis from limited data.

Findings

Enhanced inter-sentence pause distribution matching ground truth

Higher naturalness scores in subjective evaluations

Improved style consistency in synthesized speech

Abstract

Expressive speech synthesis requires vibrant prosody and well-timed pauses. We propose an effective strategy to augment a small dataset to train an expressive end-to-end Text-to-Speech model. We merge audios of emotionally congruent text using a text emotion recognizer, creating augmented expressive speech data. By training with two-sentence audio, our model learns natural breaks between lines. We further apply self-supervised contrastive training to improve the speaking style embedding extraction from speech. During inference, our model produces multi-sentence speech in one step, guided by the text-predicted speaking style. Evaluations showcase the effectiveness of our proposed approach when compared to a baseline model trained with consecutive two-sentence audio. Our synthesized speeches give a closer inter-sentence pause distribution to the ground truth speech. Subjective evaluations reveal our synthesized speech scored higher in naturalness and style suitability than the baseline.