Otem&Utem: Over- and Under-Translation Evaluation Metric for NMT

TL;DR

This paper introduces Otem and Utem, two automatic metrics for evaluating over- and under-translation in neural machine translation, showing they correlate well with human judgments and reveal limitations of traditional metrics like BLEU.

Contribution

The paper proposes novel quantitative metrics, Otem and Utem, specifically designed to assess over- and under-translation in NMT, addressing limitations of existing evaluation methods.

Findings

Otem and Utem strongly correlate with human evaluations.

They reveal inconsistencies between BLEU scores and translation quality.

Metrics effectively evaluate over- and under-translation issues.

Abstract

Although neural machine translation(NMT) yields promising translation performance, it unfortunately suffers from over- and under-translation is- sues [Tu et al., 2016], of which studies have become research hotspots in NMT. At present, these studies mainly apply the dominant automatic evaluation metrics, such as BLEU, to evaluate the overall translation quality with respect to both adequacy and uency. However, they are unable to accurately measure the ability of NMT systems in dealing with the above-mentioned issues. In this paper, we propose two quantitative metrics, the Otem and Utem, to automatically evaluate the system perfor- mance in terms of over- and under-translation respectively. Both metrics are based on the proportion of mismatched n-grams between gold ref- erence and system translation. We evaluate both metrics by comparing their scores with human evaluations, where the values of Pearson Cor- relation Coefficient reveal their strong correlation. Moreover, in-depth analyses on various translation systems indicate some inconsistency be- tween BLEU and our proposed metrics, highlighting the necessity and significance of our metrics.