PathBench: Speech Intelligibility Benchmark for Automatic Pathological Speech Assessment

TL;DR

PathBench provides a standardized benchmark for evaluating speech intelligibility assessment methods in pathological speech, facilitating fair comparison and advancing research in this field.

Contribution

It introduces PathBench, a unified benchmark with multiple protocols and datasets, and proposes DArtP, a new reference-free assessment method with high correlation.

Findings

Established benchmark baselines across six datasets.

Compared different assessment protocols and methods.

Introduced DArtP with superior correlation performance.

Abstract

Automatic speech intelligibility assessment is crucial for monitoring speech disorders and therapy efficacy. However, existing methods are difficult to compare: research is fragmented across private datasets with inconsistent protocols. We introduce PathBench, a unified benchmark for pathological speech assessment using public datasets. We compare reference-free, reference-text, and reference-audio methods across three protocols (Matched Content, Extended, and Full) representing how a linguist (controlled stimuli) versus machine learning specialist (maximum data) would approach the same data. We establish benchmark baselines across six datasets, enabling systematic evaluation of future methodological advances, and introduce Dual-ASR Articulatory Precision (DArtP), achieving the highest average correlation among reference-free methods.