On Using Very Large Target Vocabulary for Neural Machine Translation

TL;DR

This paper introduces a method using importance sampling to enable neural machine translation models to handle very large target vocabularies efficiently, improving translation quality and achieving state-of-the-art results.

Contribution

It presents a novel importance sampling approach that reduces training complexity for large vocabularies and enables efficient decoding, leading to improved translation performance.

Findings

Outperforms baseline models with small vocabularies

Achieves state-of-the-art BLEU scores on English->German translation

Almost matches top systems on English->French translation

Abstract

Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrase-based statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method that allows us to use a very large target vocabulary without increasing training complexity, based on importance sampling. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use the ensemble of a few models with very large target vocabularies, we achieve the state-of-the-art translation performance (measured by BLEU) on the English->German translation and almost as high performance as state-of-the-art English->French translation system.