The Impact of Prosodic Segmentation on Speech Synthesis of Spontaneous Speech

TL;DR

This study examines how explicit prosodic segmentation annotations, both manual and automatic, influence the naturalness and intelligibility of speech synthesized from spontaneous Brazilian Portuguese, highlighting the benefits of manual segmentation.

Contribution

It investigates the impact of explicit prosodic segmentation on spontaneous speech synthesis quality, comparing manual and automatic annotations using a non-autoregressive model.

Findings

Training with prosodic segmentation improves speech intelligibility and naturalness.

Manual segmentation introduces more variability, enhancing prosody.

Both approaches reproduce expected nuclear accent patterns, with manual aligning more closely to natural contours.

Abstract

Spontaneous speech presents several challenges for speech synthesis, particularly in capturing the natural flow of conversation, including turn-taking, pauses, and disfluencies. Although speech synthesis systems have made significant progress in generating natural and intelligible speech, primarily through architectures that implicitly model prosodic features such as pitch, intensity, and duration, the construction of datasets with explicit prosodic segmentation and their impact on spontaneous speech synthesis remains largely unexplored. This paper evaluates the effects of manual and automatic prosodic segmentation annotations in Brazilian Portuguese on the quality of speech synthesized by a non-autoregressive model, FastSpeech 2. Experimental results show that training with prosodic segmentation produced slightly more intelligible and acoustically natural speech. While automatic segmentation tends to create more regular segments, manual prosodic segmentation introduces greater variability, which contributes to more natural prosody. Analysis of neutral declarative utterances showed that both training approaches reproduced the expected nuclear accent pattern, but the prosodic model aligned more closely with natural pre-nuclear contours. To support reproducibility and future research, all datasets, source codes, and trained models are publicly available under the CC BY-NC-ND 4.0 license.