A Variational Prosody Model for Mapping the Context-Sensitive Variation of Functional Prosodic Prototypes

TL;DR

This paper introduces a Variational Prosody Model (VPM) that combines generative prosody modeling with deep learning to interpret and generate context-sensitive intonation variations, leveraging big data without specialized corpora.

Contribution

It proposes a novel VPM based on superposition of functional contours, enabling interpretable, context-sensitive prosody modeling with meaningful latent space representations.

Findings

VPM captures intrinsic variability of prosodic prototypes.

The model effectively represents multi-dimensional context variability.

VPM generates natural, dynamic prosody contours in synthesis.

Abstract

The quest for comprehensive generative models of intonation that link linguistic and paralinguistic functions to prosodic forms has been a longstanding challenge of speech communication research. Traditional intonation models have given way to the overwhelming performance of deep learning (DL) techniques for training general purpose end-to-end mappings using millions of tunable parameters. The shift towards black box machine learning models has nonetheless posed the reverse problem -- a compelling need to discover knowledge, to explain, visualise and interpret. Our work bridges between a comprehensive generative model of intonation and state-of-the-art DL techniques. We build upon the modelling paradigm of the Superposition of Functional Contours (SFC) model and propose a Variational Prosody Model (VPM) that uses a network of variational contour generators to capture the context-sensitive variation of the constituent elementary prosodic contours. We show that the VPM can give insight into the intrinsic variability of these prosodic prototypes through learning a meaningful prosodic latent space representation structure. We also show that the VPM is able to capture prosodic phenomena that have multiple dimensions of context based variability. Since it is based on the principle of superposition, the VPM does not necessitate the use of specially crafted corpora for the analysis, opening up the possibilities of using big data for prosody analysis. In a speech synthesis scenario, the model can be used to generate a dynamic and natural prosody contour that is devoid of averaging effects.