Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

TL;DR

This study examines the robustness of automatic translation evaluation metrics across unseen domains, revealing that human disagreement significantly impacts perceived metric reliability and emphasizing the importance of human benchmarks.

Contribution

The paper introduces a new multi-annotator dataset for cross-domain translation evaluation and highlights the discrepancy between metric and human agreement in unseen domains.

Findings

Metrics appear robust at segment level but are less reliable when human variation is considered.

Averaging annotations improves inter-annotator agreement by up to 0.11.

Metrics perform worse than humans on unseen chemical domains, with lower agreement scores.

Abstract

Automatic evaluation metrics are central to the development of machine translation systems, yet their robustness under domain shift remains unclear. Most metrics are developed on the Workshop on Machine Translation (WMT) benchmarks, raising concerns about their robustness to unseen domains. Prior studies that analyze unseen domains vary translation systems, annotators, or evaluation conditions, confounding domain effects with human annotation noise. To address these biases, we introduce a systematic multi-annotator Cross-Domain Error-Span-Annotation dataset (CD-ESA), comprising 18.8k human error span annotations across three language pairs, where we fix annotators within each language pair and evaluate translations of the same six translation systems across one seen news domain and two unseen technical domains. Using this dataset, we first find that automatic metrics appear surprisingly robust to domain-shifts at the segment level (up to 0.69 agreement), but this robustness largely disappears once we account for human label variation. Averaging annotations increases inter-annotator agreement by up to +0.11. Metrics struggle on the unseen chemical domain compared to humans (inter-annotator agreement of 0.78-0.83 vs. 0.96). We recommend comparing metric-human agreement against inter-annotator agreement, rather than comparing raw metric-human agreement alone, when evaluating across different domains.