Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress

TL;DR

This paper investigates whether automatic MT evaluation metrics have reached human-level performance by comparing them to human judgments, revealing that some metrics now match or surpass human agreement levels, but caution is advised in interpreting these results.

Contribution

The study introduces human baselines into MT metric evaluation, providing an upper bound and critically analyzing the implications of metrics reaching human parity.

Findings

State-of-the-art metrics often match or outperform human agreement levels.

Human annotators are not consistently superior to automatic metrics.

The results raise questions about the reliability of current evaluation methods.

Abstract

In Machine Translation (MT) evaluation, metric performance is assessed based on agreement with human judgments. In recent years, automatic metrics have demonstrated increasingly high levels of agreement with humans. To gain a clearer understanding of metric performance and establish an upper bound, we incorporate human baselines in the MT meta-evaluation, that is, the assessment of MT metrics' capabilities. Our results show that human annotators are not consistently superior to automatic metrics, with state-of-the-art metrics often ranking on par with or higher than human baselines. Despite these findings suggesting human parity, we discuss several reasons for caution. Finally, we explore the broader implications of our results for the research field, asking: Can we still reliably measure improvements in MT evaluation? With this work, we aim to shed light on the limits of our ability to measure progress in the field, fostering discussion on an issue that we believe is crucial to the entire MT evaluation community.