What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems

TL;DR

This paper investigates how humans selectively listen during spoken dialogue interactions to evaluate and improve automatic speech recognition systems in dialogue applications.

Contribution

It introduces a novel approach to evaluate ASR by analyzing human selective listening, highlighting differences between human and machine transcription capabilities.

Findings

Humans demonstrate selective listening during dialogue interactions.

Experimental results confirm differences between human and ASR transcriptions.

Proposes a new ASR evaluation method based on human selective listening.

Abstract

Spoken dialogue systems (SDSs) utilize automatic speech recognition (ASR) at the front end of their pipeline. The role of ASR in SDSs is to recognize information in user speech related to response generation appropriately. Examining selective listening of humans, which refers to the ability to focus on and listen to important parts of a conversation during the speech, will enable us to identify the ASR capabilities required for SDSs and evaluate them. In this study, we experimentally confirmed selective listening when humans generate dialogue responses by comparing human transcriptions for generating dialogue responses and reference transcriptions. Based on our experimental results, we discuss the possibility of a new ASR evaluation method that leverages human selective listening, which can identify the gap between transcription ability between ASR systems and humans.