Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation

TL;DR

This paper introduces a novel regularization method for neural machine translation that regresses word and sentence embeddings during training, significantly enhancing translation quality especially in low-resource scenarios.

Contribution

It proposes regressing word and sentence embeddings as a new regularization technique to improve NMT generalization, particularly for limited data settings.

Findings

ReWE and ReSE outperform state-of-the-art baselines in multiple language pairs.

The methods yield up to +5.15 BLEU points improvement in low-resource Basque-English translation.

Regularizers improve word clustering and translation accuracy.

Abstract

In recent years, neural machine translation (NMT) has become the dominant approach in automated translation. However, like many other deep learning approaches, NMT suffers from overfitting when the amount of training data is limited. This is a serious issue for low-resource language pairs and many specialized translation domains that are inherently limited in the amount of available supervised data. For this reason, in this paper we propose regressing word (ReWE) and sentence (ReSE) embeddings at training time as a way to regularize NMT models and improve their generalization. During training, our models are trained to jointly predict categorical (words in the vocabulary) and continuous (word and sentence embeddings) outputs. An extensive set of experiments over four language pairs of variable training set size has showed that ReWE and ReSE can outperform strong state-of-the-art baseline models, with an improvement that is larger for smaller training sets (e.g., up to +5:15 BLEU points in Basque-English translation). Visualizations of the decoder's output space show that the proposed regularizers improve the clustering of unique words, facilitating correct predictions. In a final experiment on unsupervised NMT, we show that ReWE and ReSE are also able to improve the quality of machine translation when no parallel data are available.