Machine Translation Evaluation using Bi-directional Entailment

TL;DR

This paper introduces a novel machine translation evaluation metric based on bi-directional entailment using deep learning, which correlates better with human judgment than traditional n-gram overlap metrics.

Contribution

The paper presents a new semantic similarity metric for MT evaluation using BERT-based entailment in both directions, improving correlation with human scores.

Findings

Our metric outperforms BLEU and METEOR in correlation with human judgments.

The approach effectively captures paraphrasing and semantic equivalence.

Evaluation on WMT datasets demonstrates its robustness.

Abstract

In this paper, we propose a new metric for Machine Translation (MT) evaluation, based on bi-directional entailment. We show that machine generated translation can be evaluated by determining paraphrasing with a reference translation provided by a human translator. We hypothesize, and show through experiments, that paraphrasing can be detected by evaluating entailment relationship in the forward and backward direction. Unlike conventional metrics, like BLEU or METEOR, our approach uses deep learning to determine the semantic similarity between candidate and reference translation for generating scores rather than relying upon simple n-gram overlap. We use BERT's pre-trained implementation of transformer networks, fine-tuned on MNLI corpus, for natural language inferencing. We apply our evaluation metric on WMT'14 and WMT'17 dataset to evaluate systems participating in the translation task and find that our metric has a better correlation with the human annotated score compared to the other traditional metrics at system level.