Attention-based Vocabulary Selection for NMT Decoding

TL;DR

This paper introduces a novel method for dynamically learning candidate vocabularies for NMT decoding directly from the attention mechanism, significantly speeding up translation without quality loss.

Contribution

It proposes a simple, novel approach to generate optimized candidate lists from the attention layer during training, eliminating the need for external candidate computation.

Findings

Significant decoding speedup achieved

No loss in translation quality observed

Applicable to multiple language pairs

Abstract

Neural Machine Translation (NMT) models usually use large target vocabulary sizes to capture most of the words in the target language. The vocabulary size is a big factor when decoding new sentences as the final softmax layer normalizes over all possible target words. To address this problem, it is widely common to restrict the target vocabulary with candidate lists based on the source sentence. Usually, the candidate lists are a combination of external word-to-word aligner, phrase table entries or most frequent words. In this work, we propose a simple and yet novel approach to learn candidate lists directly from the attention layer during NMT training. The candidate lists are highly optimized for the current NMT model and do not need any external computation of the candidate pool. We show significant decoding speedup compared with using the entire vocabulary, without losing any translation quality for two language pairs.