Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation

TL;DR

This paper introduces new benchmark datasets and evaluation methods for assessing discourse phenomena in machine translation, revealing that current context-aware models do not consistently improve discourse-related translation quality.

Contribution

It presents the first benchmark datasets and evaluation methods targeting discourse phenomena in MT, and evaluates baseline models revealing their limited improvements.

Findings

Existing models do not consistently improve discourse translation.

Benchmark datasets reveal gaps in current context-aware MT.

Evaluation methods highlight the need for better discourse modeling.

Abstract

Despite increasing instances of machine translation (MT) systems including contextual information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark datasets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several baseline MT systems on the curated datasets. Surprisingly, we find that existing context-aware models do not improve discourse-related translations consistently across languages and phenomena.