Parallel Attention Mechanisms in Neural Machine Translation

TL;DR

This paper introduces a parallel attention mechanism in neural machine translation, reducing training time and improving translation quality by modifying the Transformer architecture to run attention modules in parallel.

Contribution

It proposes a novel parallel attention approach within the Transformer model, significantly decreasing training time and enhancing translation performance.

Findings

Reduced training time compared to standard Transformer

Achieved state-of-the-art BLEU scores on English-German and English-French tasks

Demonstrated effectiveness of parallel attention modules in NMT

Abstract

Recent papers in neural machine translation have proposed the strict use of attention mechanisms over previous standards such as recurrent and convolutional neural networks (RNNs and CNNs). We propose that by running traditionally stacked encoding branches from encoder-decoder attention- focused architectures in parallel, that even more sequential operations can be removed from the model and thereby decrease training time. In particular, we modify the recently published attention-based architecture called Transformer by Google, by replacing sequential attention modules with parallel ones, reducing the amount of training time and substantially improving BLEU scores at the same time. Experiments over the English to German and English to French translation tasks show that our model establishes a new state of the art.