Rethinking Round-Trip Translation for Machine Translation Evaluation

TL;DR

This paper revisits round-trip translation for machine translation evaluation, revealing its potential to assess translation quality without references and improving various evaluation tasks by addressing previous misunderstandings caused by copying mechanisms.

Contribution

The study uncovers that removing copying mechanisms in SMT clarifies round-trip translation's effectiveness, enabling its use in reference-free evaluation and enhancing multiple MT assessment methods.

Findings

Round-trip translation correlates well with forward translation performance after removing copying mechanisms.

Round-trip translation can predict forward translation scores without references.

It improves quality estimation models and helps identify adversarial systems.

Abstract

Automatic evaluation on low-resource language translation suffers from a deficiency of parallel corpora. Round-trip translation could be served as a clever and straightforward technique to alleviate the requirement of the parallel evaluation corpus. However, there was an observation of obscure correlations between the evaluation scores by forward and round-trip translations in the era of statistical machine translation (SMT). In this paper, we report the surprising finding that round-trip translation can be used for automatic evaluation without the references. Firstly, our revisit on the round-trip translation in SMT evaluation unveils that its long-standing misunderstanding is essentially caused by copying mechanism. After removing copying mechanism in SMT, round-trip translation scores can appropriately reflect the forward translation performance. Then, we demonstrate the rectification is overdue as round-trip translation could benefit multiple machine translation evaluation tasks. To be more specific, round-trip translation could be used i) to predict corresponding forward translation scores; ii) to improve the performance of the recently advanced quality estimation model; and iii) to identify adversarial competitors in shared tasks via cross-system verification.