Evaluating Automatic Metrics with Incremental Machine Translation Systems

TL;DR

This paper presents a large, longitudinal dataset of commercial machine translations to evaluate the effectiveness of automatic MT metrics over time, highlighting neural metrics' superiority and the impact of translation quality on metric reliability.

Contribution

Introduces a six-year, multi-language dataset for evaluating MT metrics and investigates how translation quality influences metric performance, addressing limitations of smaller datasets.

Findings

Neural metrics outperform non-neural metrics.

Translation quality significantly affects metric reliability.

Dataset serves as a valuable benchmark for MT metric evaluation.

Abstract

We introduce a dataset comprising commercial machine translations, gathered weekly over six years across 12 translation directions. Since human A/B testing is commonly used, we assume commercial systems improve over time, which enables us to evaluate machine translation (MT) metrics based on their preference for more recent translations. Our study not only confirms several prior findings, such as the advantage of neural metrics over non-neural ones, but also explores the debated issue of how MT quality affects metric reliability--an investigation that smaller datasets in previous research could not sufficiently explore. Overall, our research demonstrates the dataset's value as a testbed for metric evaluation. We release our code at https://github.com/gjwubyron/Evo