Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translation

TL;DR

This study evaluates how the rise of Large Language Models affects existing methods for predicting machine translation quality, using a multi-candidate dataset and correlation metrics to compare traditional and new paradigms.

Contribution

It provides insights into the changing effectiveness of quality prediction paradigms in the context of LLM integration into MT workflows.

Findings

LLMs impact the reliability of traditional quality prediction methods

Document-level translation challenges are reduced with LLMs

Candidate-side QE remains valuable for quality assessment

Abstract

This paper investigates two complementary paradigms for predicting machine translation (MT) quality: source-side difficulty prediction and candidate-side quality estimation (QE). The rapid adoption of Large Language Models (LLMs) into MT workflows is reshaping the research landscape, yet its impact on established quality prediction paradigms remains underexplored. We study this issue through a series of "hindsight" experiments on a unique, multi-candidate dataset resulting from a genuine MT post-editing (MTPE) project. The dataset consists of over 6,000 English source segments with nine translation hypotheses from a diverse set of traditional neural MT systems and advanced LLMs, all evaluated against a single, final human post-edited reference. Using Kendall's rank correlation, we assess the predictive power of source-side difficulty metrics, candidate-side QE models and position heuristics against two gold-standard scores: TER (as a proxy for post-editing effort) and COMET (as a proxy for human judgment). Our findings highlight that the architectural shift towards LLMs alters the reliability of established quality prediction methods while simultaneously mitigating previous challenges in document-level translation.