HATS: An Open data set Integrating Human Perception Applied to the Evaluation of Automatic Speech Recognition Metrics

TL;DR

This paper introduces HATS, a French dataset with human perception annotations of ASR errors, and analyzes how well different metrics align with human judgments.

Contribution

The paper presents HATS, a novel human-annotated dataset for evaluating ASR transcriptions, and examines the correlation between metrics and human preferences.

Findings

Embedding-based metrics correlate better with human preferences than traditional metrics.

HATS provides a new resource for evaluating ASR systems from a human perception perspective.

Analysis reveals limitations of existing metrics in capturing human judgments.

Abstract

Conventionally, Automatic Speech Recognition (ASR) systems are evaluated on their ability to correctly recognize each word contained in a speech signal. In this context, the word error rate (WER) metric is the reference for evaluating speech transcripts. Several studies have shown that this measure is too limited to correctly evaluate an ASR system, which has led to the proposal of other variants of metrics (weighted WER, BERTscore, semantic distance, etc.). However, they remain system-oriented, even when transcripts are intended for humans. In this paper, we firstly present Human Assessed Transcription Side-by-side (HATS), an original French manually annotated data set in terms of human perception of transcription errors produced by various ASR systems. 143 humans were asked to choose the best automatic transcription out of two hypotheses. We investigated the relationship between human preferences and various ASR evaluation metrics, including lexical and embedding-based ones, the latter being those that correlate supposedly the most with human perception.