Measuring and Increasing Context Usage in Context-Aware Machine Translation

TL;DR

This paper introduces a metric to measure context utilization in neural machine translation, and proposes a training method to enhance context usage, resulting in improved translation quality and coherence.

Contribution

It presents a new metric for quantifying context usage and a simple training technique to increase context utilization in document-level translation models.

Findings

Target context is referenced more than source context.

Longer context conditioning has diminishing returns.

Enhanced context usage improves translation quality and coherence.

Abstract

Recent work in neural machine translation has demonstrated both the necessity and feasibility of using inter-sentential context -- context from sentences other than those currently being translated. However, while many current methods present model architectures that theoretically can use this extra context, it is often not clear how much they do actually utilize it at translation time. In this paper, we introduce a new metric, conditional cross-mutual information, to quantify the usage of context by these models. Using this metric, we measure how much document-level machine translation systems use particular varieties of context. We find that target context is referenced more than source context, and that conditioning on a longer context has a diminishing effect on results. We then introduce a new, simple training method, context-aware word dropout, to increase the usage of context by context-aware models. Experiments show that our method increases context usage and that this reflects on the translation quality according to metrics such as BLEU and COMET, as well as performance on anaphoric pronoun resolution and lexical cohesion contrastive datasets.