Is Word Error Rate a good evaluation metric for Speech Recognition in Indic Languages?

TL;DR

This paper introduces new error metrics, AWER and ACER, tailored for Indic languages with complex scripts, demonstrating improved interpretability and accuracy over traditional WER and CER in Hindi speech recognition.

Contribution

The paper proposes and validates new error metrics, AWER and ACER, specifically designed for Indic languages with complex character forms, enhancing error analysis in speech recognition systems.

Findings

AWER and ACER outperform WER and CER in Hindi speech recognition

Significant improvements in error interpretability up to 3% in AWER and 7% in ACER

Open source dataset for Hindi with new metrics and scripts

Abstract

We propose a new method for the calculation of error rates in Automatic Speech Recognition (ASR). This new metric is for languages that contain half characters and where the same character can be written in different forms. We implement our methodology in Hindi which is one of the main languages from Indic context and we think this approach is scalable to other similar languages containing a large character set. We call our metrics Alternate Word Error Rate (AWER) and Alternate Character Error Rate (ACER). We train our ASR models using wav2vec 2.0\cite{baevski2020wav2vec} for Indic languages. Additionally we use language models to improve our model performance. Our results show a significant improvement in analyzing the error rates at word and character level and the interpretability of the ASR system is improved upto $3$\% in AWER and $7$\% in ACER for Hindi. Our experiments suggest that in languages which have complex pronunciation, there are multiple ways of writing words without changing their meaning. In such cases AWER and ACER will be more useful rather than WER and CER as metrics. Further, we open source a new benchmarking dataset of 21 hours for Hindi with the new metric scripts.