Emphasis Sensitivity in Speech Representations

TL;DR

This paper explores how modern speech models encode prosodic emphasis, revealing that emphasis is represented as a structured, low-dimensional transformation that becomes more pronounced with task-specific fine-tuning.

Contribution

It introduces a residual-based framework to analyze emphasis sensitivity, demonstrating that emphasis encoding is relational, structured, and more compact in fine-tuned models.

Findings

Residuals correlate with duration changes

Residuals perform poorly at word identity prediction

Emphasis encoding becomes more structured after fine-tuning

Abstract

This work investigates whether modern speech models are sensitive to prosodic emphasis - whether they encode emphasized and neutral words in systematically different ways. Prior work typically relies on isolated acoustic correlates (e.g., pitch, duration) or label prediction, both of which miss the relational structure of emphasis. This paper proposes a residual-based framework, defining emphasis as the difference between paired neutral and emphasized word representations. Analysis on self-supervised speech models shows that these residuals correlate strongly with duration changes and perform poorly at word identity prediction, indicating a structured, relational encoding of prosodic emphasis. In ASR fine-tuned models, residuals occupy a subspace up to 50% more compact than in pre-trained models, further suggesting that emphasis is encoded as a consistent, low-dimensional transformation that becomes more structured with task-specific learning.