Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus

TL;DR

This paper introduces a richly annotated parallel corpus to evaluate discourse phenomena in machine translation, revealing fundamental differences between MT outputs and human translations at the discourse level.

Contribution

It presents a new discourse-annotated dataset based on BWB, enabling systematic analysis of discourse structure challenges in document-level machine translation.

Findings

MT outputs differ from human translations in discourse structure

Discourse phenomena like entity, coreference, and quotation are crucial for evaluation

The dataset facilitates future research in context-aware MT

Abstract

Several recent papers claim human parity at sentence-level Machine Translation (MT), especially in high-resource languages. Thus, in response, the MT community has, in part, shifted its focus to document-level translation. Translating documents requires a deeper understanding of the structure and meaning of text, which is often captured by various kinds of discourse phenomena such as consistency, coherence, and cohesion. However, this renders conventional sentence-level MT evaluation benchmarks inadequate for evaluating the performance of context-aware MT systems. This paper presents a new dataset with rich discourse annotations, built upon the large-scale parallel corpus BWB introduced in Jiang et al. (2022). The new BWB annotation introduces four extra evaluation aspects, i.e., entity, terminology, coreference, and quotation, covering 15,095 entity mentions in both languages. Using these annotations, we systematically investigate the similarities and differences between the discourse structures of source and target languages, and the challenges they pose to MT. We discover that MT outputs differ fundamentally from human translations in terms of their latent discourse structures. This gives us a new perspective on the challenges and opportunities in document-level MT. We make our resource publicly available to spur future research in document-level MT and the generalization to other language translation tasks.