Improving Multilingual Translation by Representation and Gradient Regularization

TL;DR

This paper introduces a joint regularization method at representation and gradient levels to improve multilingual NMT, significantly reducing off-target translations and enhancing zero-shot translation quality.

Contribution

It proposes a novel combined regularization approach that leverages auxiliary tasks and limited direct data to improve multilingual translation models.

Findings

Reduces off-target translation by a significant margin.

Improves zero-shot translation BLEU scores by over 5 and 10 points.

Effective even without access to direct data.

Abstract

Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations -- commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regularize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language prediction task to regularize decoder outputs to retain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our method also works well when the small amount of direct data is not available.