Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview

TL;DR

This paper provides a comprehensive overview of the evolution of machine translation evaluation methods, including traditional, automatic, and recent meta-evaluation techniques, emphasizing the role of large language models and statistical confidence in assessment accuracy.

Contribution

It offers a systematic, up-to-date overview of MT evaluation methods, highlighting recent advances with large language models and statistical confidence estimation for human evaluation.

Findings

Large language models improve automatic metric customization.

Meta-evaluation techniques enhance assessment reliability.

Statistical confidence estimation informs human evaluation sample sizes.

Abstract

Starting from the 1950s, Machine Translation (MT) was challenged by different scientific solutions, which included rule-based methods, example-based and statistical models (SMT), to hybrid models, and very recent years the neural models (NMT). While NMT has achieved a huge quality improvement in comparison to conventional methodologies, by taking advantage of a huge amount of parallel corpora available from the internet and the recently developed super computational power support with an acceptable cost, it struggles to achieve real human parity in many domains and most language pairs, if not all of them. Alongside the long road of MT research and development, quality evaluation metrics played very important roles in MT advancement and evolution. In this tutorial, we overview the traditional human judgement criteria, automatic evaluation metrics, unsupervised quality estimation models, as well as the meta-evaluation of the evaluation methods. Among these, we will also cover the very recent work in the MT evaluation (MTE) fields, taking advantage of the large size of pre-trained language models for automatic metric customisation towards exactly deployed language pairs and domains. In addition, we also introduce the statistical confidence estimation regarding the sample size needed for human evaluation in real practice simulation. Full tutorial material is \textbf{available} to download at https://github.com/poethan/LREC22_MetaEval_Tutorial.