What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels

TL;DR

This paper introduces an information-theoretic method using large speech and language models to quantify how much meaning, such as emotion or sarcasm, is conveyed by prosody versus text in speech communication.

Contribution

It presents a novel approach to measure the information carried by prosody independently from text, revealing the significant role of prosody in conveying certain types of meaning.

Findings

Prosody transmits over ten times more information about sarcasm and emotion than text.

Prosody provides less additional information about questionhood compared to sarcasm and emotion.

The approach can be extended to analyze other dimensions of meaning and communication channels.

Abstract

Prosody -- the melody of speech -- conveys critical information often not captured by the words or text of a message. In this paper, we propose an information-theoretic approach to quantify how much information is expressed by prosody alone and not by text, and crucially, what that information is about. Our approach applies large speech and language models to estimate the mutual information between a particular dimension of an utterance's meaning (e.g., its emotion) and any of its communication channels (e.g., audio or text). We then use this approach to quantify how much information is conveyed by audio and text about sarcasm, emotion, and questionhood, using speech from television and podcasts. We find that for sarcasm and emotion the audio channel -- and by implication the prosodic channel -- transmits over an order of magnitude more information about these features than the text channel alone, at least when long-term context beyond the current sentence is unavailable. For questionhood, prosody provides comparatively less additional information. We conclude by outlining a program applying our approach to more dimensions of meaning, communication channels, and languages.