Aligning ASR Evaluation with Human and LLM Judgments: Intelligibility Metrics Using Phonetic, Semantic, and NLI Approaches

TL;DR

This paper introduces a new ASR evaluation metric that combines phonetic, semantic, and NLI approaches to better reflect human judgments of intelligibility, especially for speech impairments.

Contribution

The paper proposes a novel metric integrating NLI, semantic, and phonetic similarity, improving correlation with human intelligibility judgments over traditional error-based metrics.

Findings

Achieves 0.890 correlation with human judgments

Outperforms traditional metrics like WER and CER

Highlights importance of semantic and phonetic factors in ASR evaluation

Abstract

Traditional ASR metrics like WER and CER fail to capture intelligibility, especially for dysarthric and dysphonic speech, where semantic alignment matters more than exact word matches. ASR systems struggle with these speech types, often producing errors like phoneme repetitions and imprecise consonants, yet the meaning remains clear to human listeners. We identify two key challenges: (1) Existing metrics do not adequately reflect intelligibility, and (2) while LLMs can refine ASR output, their effectiveness in correcting ASR transcripts of dysarthric speech remains underexplored. To address this, we propose a novel metric integrating Natural Language Inference (NLI) scores, semantic similarity, and phonetic similarity. Our ASR evaluation metric achieves a 0.890 correlation with human judgments on Speech Accessibility Project data, surpassing traditional methods and emphasizing the need to prioritize intelligibility over error-based measures.