IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages

TL;DR

This paper introduces IndicMT Eval, a dataset for evaluating machine translation metrics for Indian languages, revealing that current metrics like COMET correlate well with human judgments but often miss fluency errors.

Contribution

The paper creates a new MQM dataset with 7000 annotations for Indian languages and systematically evaluates existing metrics' effectiveness in this context.

Findings

Pre-trained metrics like COMET show high correlation with human scores.

Current metrics inadequately capture fluency errors in Indian languages.

The dataset facilitates future research in MT evaluation for Indian languages.

Abstract

The rapid growth of machine translation (MT) systems has necessitated comprehensive studies to meta-evaluate evaluation metrics being used, which enables a better selection of metrics that best reflect MT quality. Unfortunately, most of the research focuses on high-resource languages, mainly English, the observations for which may not always apply to other languages. Indian languages, having over a billion speakers, are linguistically different from English, and to date, there has not been a systematic study of evaluating MT systems from English into Indian languages. In this paper, we fill this gap by creating an MQM dataset consisting of 7000 fine-grained annotations, spanning 5 Indian languages and 7 MT systems, and use it to establish correlations between annotator scores and scores obtained using existing automatic metrics. Our results show that pre-trained metrics, such as COMET, have the highest correlations with annotator scores. Additionally, we find that the metrics do not adequately capture fluency-based errors in Indian languages, and there is a need to develop metrics focused on Indian languages. We hope that our dataset and analysis will help promote further research in this area.