Improving Robustness of Retrieval Augmented Translation via Shuffling of Suggestions

TL;DR

This paper investigates the impact of domain mismatch in retrieval-augmented neural machine translation and proposes a training method involving suggestion shuffling to improve robustness against domain differences.

Contribution

It introduces a simple training technique with suggestion shuffling that enhances the model's tolerance to domain mismatched translation memories in retrieval-augmented NMT.

Findings

Regains up to 5.8 BLEU points with domain-mismatched TMs

Maintains competitiveness with relevant TMs

Improves robustness of retrieval-augmented translation systems

Abstract

Several recent studies have reported dramatic performance improvements in neural machine translation (NMT) by augmenting translation at inference time with fuzzy-matches retrieved from a translation memory (TM). However, these studies all operate under the assumption that the TMs available at test time are highly relevant to the testset. We demonstrate that for existing retrieval augmented translation methods, using a TM with a domain mismatch to the test set can result in substantially worse performance compared to not using a TM at all. We propose a simple method to expose fuzzy-match NMT systems during training and show that it results in a system that is much more tolerant (regaining up to 5.8 BLEU) to inference with TMs with domain mismatch. Also, the model is still competitive to the baseline when fed with suggestions from relevant TMs.