The Role of Prosody in Spoken Question Answering

TL;DR

This paper explores how prosody, the intonation and rhythm in speech, can enhance spoken question answering systems, revealing that prosody alone can be informative but is often overshadowed by lexical cues.

Contribution

It demonstrates the importance of prosody in spoken QA and highlights the need for better integration methods to leverage prosodic information effectively.

Findings

Models trained on prosody alone perform reasonably well.

Lexical information dominates when available.

Prosody provides valuable supplementary cues.

Abstract

Spoken language understanding research to date has generally carried a heavy text perspective. Most datasets are derived from text, which is then subsequently synthesized into speech, and most models typically rely on automatic transcriptions of speech. This is to the detriment of prosody--additional information carried by the speech signal beyond the phonetics of the words themselves and difficult to recover from text alone. In this work, we investigate the role of prosody in Spoken Question Answering. By isolating prosodic and lexical information on the SLUE-SQA-5 dataset, which consists of natural speech, we demonstrate that models trained on prosodic information alone can perform reasonably well by utilizing prosodic cues. However, we find that when lexical information is available, models tend to predominantly rely on it. Our findings suggest that while prosodic cues provide valuable supplementary information, more effective integration methods are required to ensure prosody contributes more significantly alongside lexical features.