A Benchmark of French ASR Systems Based on Error Severity

TL;DR

This paper introduces a new error severity-based evaluation metric for French ASR systems, providing a more human-centric assessment of transcription quality beyond traditional metrics like WER.

Contribution

It proposes a novel error categorization method based on linguistic and contextual criteria and benchmarks 10 state-of-the-art French ASR systems using this approach.

Findings

Identifies which ASR systems offer the most understandable transcriptions.

Highlights strengths and weaknesses of different ASR models.

Provides insights into error types affecting user comprehension.

Abstract

Automatic Speech Recognition (ASR) transcription errors are commonly assessed using metrics that compare them with a reference transcription, such as Word Error Rate (WER), which measures spelling deviations from the reference, or semantic score-based metrics. However, these approaches often overlook what is understandable to humans when interpreting transcription errors. To address this limitation, a new evaluation is proposed that categorizes errors into four levels of severity, further divided into subtypes, based on objective linguistic criteria, contextual patterns, and the use of content words as the unit of analysis. This metric is applied to a benchmark of 10 state-of-the-art ASR systems on French language, encompassing both HMM-based and end-to-end models. Our findings reveal the strengths and weaknesses of each system, identifying those that provide the most comfortable reading experience for users.