Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosody

TL;DR

This study explores how word surprisal from large language models correlates with word prominence and its potential to improve speech synthesis prosody, finding limited benefits from using surprisal features.

Contribution

It investigates the relationship between LLM-derived surprisal and speech prominence, analyzing effects of context length and model size on prosody modeling.

Findings

Word surprisal and prominence are moderately correlated.

Longer context and larger LLMs tend to underpredict prominence.

Conditioning speech synthesis on surprisal yields minimal improvements.

Abstract

This paper investigates the use of word surprisal, a measure of the predictability of a word in a given context, as a feature to aid speech synthesis prosody. We explore how word surprisal extracted from large language models (LLMs) correlates with word prominence, a signal-based measure of the salience of a word in a given discourse. We also examine how context length and LLM size affect the results, and how a speech synthesizer conditioned with surprisal values compares with a baseline system. To evaluate these factors, we conducted experiments using a large corpus of English text and LLMs of varying sizes. Our results show that word surprisal and word prominence are moderately correlated, suggesting that they capture related but distinct aspects of language use. We find that length of context and size of the LLM impact the correlations, but not in the direction anticipated, with longer contexts and larger LLMs generally underpredicting prominent words in a nearly linear manner. We demonstrate that, in line with these findings, a speech synthesizer conditioned with surprisal values provides a minimal improvement over the baseline with the results suggesting a limited effect of using surprisal values for eliciting appropriate prominence patterns.