Improving Lexical Choice in Neural Machine Translation

TL;DR

This paper proposes two methods to improve neural machine translation by addressing rare word mistranslation, including normalizing output vector norms and adding a lexical module, resulting in significant BLEU score improvements across multiple language pairs.

Contribution

It introduces norm fixing of output vectors and a lexical module to enhance rare word translation in neural machine translation models.

Findings

Up to +4.3 BLEU score improvement

Outperforms phrase-based translation in most settings

Effective across diverse language pairs and data sizes

Abstract

We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings.