Towards explainable reference-free speech intelligibility evaluation of people with pathological speech

TL;DR

This paper introduces an explainable, reference-free speech intelligibility metric called the ASR Inconsistency Score, which correlates well with expert ratings and offers a more interpretable alternative to traditional reference-based methods.

Contribution

The work presents a novel, explainable, reference-free speech intelligibility measure that performs comparably or better than standard reference-based metrics across multiple languages.

Findings

High correlation with expert perceptual ratings

Performance matches or exceeds reference-based WER baseline

Effective across Dutch, Spanish, and English pathological speech

Abstract

Objective assessment of speech that reflects meaningful changes in communication is crucial for clinical decision making and reproducible research. While existing objective assessments, particularly reference-based approaches, can capture intelligibility changes, they are often hindered by lack of explainability and the need for labor-intensive manual transcriptions. To address these issues, this work proposes the reference-free, explainable ASR Inconsistency Score. We evaluate this method on pathological speech in Dutch, Spanish and English, and compare its performance to a reference-based Word Error Rate (WER) baseline. Our results demonstrate that the ASR Inconsistency Score achieves a high correlation with expert perceptual ratings, with performance closely matching, and in one case exceeding, a standard reference-based Word Error Rate (WER) baseline.