Using Confidence Scores to Improve Eyes-free Detection of Speech Recognition Errors

TL;DR

This paper explores enhancing eyes-free detection of speech recognition errors by adjusting audio playback based on confidence scores, leading to improved error detection and efficiency for users, especially those with visual impairments.

Contribution

It introduces a novel method of manipulating audio output according to confidence levels to improve error detection without visual cues.

Findings

Selective slowing of audio increased error detection by 12%.

It reduced listening and decision time by 11%.

Confidence-based audio manipulation benefits eyes-free error detection.

Abstract

Conversational systems rely heavily on speech recognition to interpret and respond to user commands and queries. Despite progress on speech recognition accuracy, errors may still sometimes occur and can significantly affect the end-user utility of such systems. While visual feedback can help detect errors, it may not always be practical, especially for people who are blind or low-vision. In this study, we investigate ways to improve error detection by manipulating the audio output of the transcribed text based on the recognizer's confidence level in its result. Our findings show that selectively slowing down the audio when the recognizer exhibited uncertainty led to a 12% relative increase in participants' ability to detect errors compared to uniformly slowing the audio. It also reduced the time it took participants to listen to the recognition result and decide if there was an error by 11%.