Lost in Transcription: Subtitle Errors in Automatic Speech Recognition Reduce Speaker and Content Evaluations

TL;DR

This study shows that subtitle errors in automatic speech recognition systems negatively impact viewers' evaluations of speakers and content, with potential bias against speakers with accents.

Contribution

It provides empirical evidence on how subtitle errors influence perceptions and highlights the potential for accent-based bias in viewer evaluations.

Findings

Error-prone subtitles reduce speaker evaluations

Error-prone subtitles reduce content evaluations

No significant difference between accent groups when controlling for subtitle quality

Abstract

Researchers have demonstrated that Automatic Speech Recognition (ASR) systems perform differently across demographic groups. In this work, we examined how subtitle errors affect evaluations of speakers and their content using a preregistered online experiment (N=207, U.S.-based crowdworkers). Participants watched speakers with various accents deliver a talk in which the subtitles were accurate or error-prone. Our results indicate that error-prone subtitles consistently reduce both speaker and content evaluations for all speakers. We did not see disparate impact between the accent groups, controlling for subtitle quality. Taken together, though, the findings of this short paper imply that speakers with accents for which ASR systems perform poorly are likely to be further penalized by viewers with lower evaluations.