Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?

TL;DR

This paper investigates whether speech-to-text translation systems effectively utilize prosody features like intonation and rhythm, introducing a new benchmark and methodology to evaluate prosody awareness in translation models.

Contribution

The authors propose a novel evaluation methodology and benchmark (ContraProST) for assessing prosody awareness in speech-to-text translation systems, and analyze how different models leverage prosodic information.

Findings

S2TT models have some internal prosody representation.

End-to-end systems outperform cascaded systems in prosody utilization.

Cascaded systems can capture prosody but less effectively depending on transcript surface form.

Abstract

The prosody of a spoken utterance, including features like stress, intonation and rhythm, can significantly affect the underlying semantics, and as a consequence can also affect its textual translation. Nevertheless, prosody is rarely studied within the context of speech-to-text translation (S2TT) systems. In particular, end-to-end (E2E) systems have been proposed as well-suited for prosody-aware translation because they have direct access to the speech signal when making translation decisions, but the understanding of whether this is successful in practice is still limited. A main challenge is the difficulty of evaluating prosody awareness in translation. To address this challenge, we introduce an evaluation methodology and a focused benchmark (named ContraProST) aimed at capturing a wide range of prosodic phenomena. Our methodology uses large language models and controllable text-to-speech (TTS) to generate contrastive examples. Through experiments in translating English speech into German, Spanish, and Japanese, we find that (a) S2TT models possess some internal representation of prosody, but the prosody signal is often not strong enough to affect the translations, (b) E2E systems outperform cascades of speech recognition and text translation systems, confirming their theoretical advantage in this regard, and (c) certain cascaded systems also capture prosodic information in the translation, but only to a lesser extent that depends on the particulars of the transcript's surface form.