Grammar Accuracy Evaluation (GAE): Quantifiable Quantitative Evaluation of Machine Translation Models

TL;DR

This paper introduces Grammar Accuracy Evaluation (GAE), a new quantifiable method for evaluating machine translation quality that addresses the limitations of traditional metrics like BLEU by providing more flexible and specific grammatical assessments.

Contribution

The paper proposes GAE, a novel evaluation metric that offers a more reliable and detailed grammatical assessment of machine translation outputs compared to existing methods.

Findings

GAE provides more flexible evaluation of synonyms and sentence structures.

BLEU score does not fully reflect the absolute performance of translation models.

GAE compensates for BLEU's shortcomings by offering specific grammatical criteria.

Abstract

Natural Language Generation (NLG) refers to the operation of expressing the calculation results of a system in human language. Since the quality of generated sentences from an NLG model cannot be fully represented using only quantitative evaluation, they are evaluated using qualitative evaluation by humans in which the meaning or grammar of a sentence is scored according to a subjective criterion. Nevertheless, the existing evaluation methods have a problem as a large score deviation occurs depending on the criteria of evaluators. In this paper, we propose Grammar Accuracy Evaluation (GAE) that can provide the specific evaluating criteria. As a result of analyzing the quality of machine translation by BLEU and GAE, it was confirmed that the BLEU score does not represent the absolute performance of machine translation models and GAE compensates for the shortcomings of BLEU with flexible evaluation of alternative synonyms and changes in sentence structure.