Investigating Length Issues in Document-level Machine Translation

TL;DR

This paper examines how increasing document length impacts machine translation quality, revealing that longer texts and sentence positions within documents significantly affect translation performance, with current methods still lagging behind sentence-level translation.

Contribution

The study provides a new measurement approach for length effects in document-level MT and demonstrates the persistent challenges with longer texts and sentence positioning.

Findings

Translation quality decreases as input length increases.

Sentence position within a document influences translation accuracy.

Manipulating document length distribution and positional embeddings has limited effect.

Abstract

Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we challenge the ability of MT systems to handle texts comprising up to several thousands of tokens. We design and implement a new approach designed to precisely measure the effect of length increments on MT outputs. Our experiments with two representative architectures unambiguously show that (a)~translation performance decreases with the length of the input text; (b)~the position of sentences within the document matters, and translation quality is higher for sentences occurring earlier in a document. We further show that manipulating the distribution of document lengths and of positional embeddings only marginally mitigates such problems. Our results suggest that even though document-level MT is computationally feasible, it does not yet match the performance of sentence-based MT.