A Large-Scale Test Set for the Evaluation of Context-Aware Pronoun Translation in Neural Machine Translation

TL;DR

This paper introduces a large-scale contrastive test set to evaluate how well neural machine translation models handle pronouns across sentence boundaries, highlighting the importance of context for accurate translation.

Contribution

It presents a specialized test suite for pronoun translation evaluation and demonstrates the effectiveness of context-aware models and parameter tying techniques.

Findings

Context-aware models outperform baselines in pronoun translation accuracy.

BLEU scores show moderate improvements, but contrastive accuracy reveals larger gains.

Parameter tying enhances multi-encoder model performance.

Abstract

The translation of pronouns presents a special challenge to machine translation to this day, since it often requires context outside the current sentence. Recent work on models that have access to information across sentence boundaries has seen only moderate improvements in terms of automatic evaluation metrics such as BLEU. However, metrics that quantify the overall translation quality are ill-equipped to measure gains from additional context. We argue that a different kind of evaluation is needed to assess how well models translate inter-sentential phenomena such as pronouns. This paper therefore presents a test suite of contrastive translations focused specifically on the translation of pronouns. Furthermore, we perform experiments with several context-aware models. We show that, while gains in BLEU are moderate for those systems, they outperform baselines by a large margin in terms of accuracy on our contrastive test set. Our experiments also show the effectiveness of parameter tying for multi-encoder architectures.