Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages

TL;DR

This paper introduces an orthographically-informed evaluation framework for Indian language speech recognition, improving accuracy by accounting for spelling variations and aligning better with human perception.

Contribution

It proposes a novel benchmark leveraging LLMs to capture permissible orthographic variations, enhancing the evaluation of ASR systems for Indian languages.

Findings

OIWER reduces error rates by 6.3 points on average

It narrows model performance gaps significantly

Aligns evaluation metrics more closely with human perception

Abstract

Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.