A Convolutional Encoder Model for Neural Machine Translation

TL;DR

This paper introduces a convolutional encoder architecture for neural machine translation that is faster and simpler than traditional LSTM-based models, achieving comparable or better accuracy and significantly improved decoding speed.

Contribution

The paper presents a convolutional encoder model that enables simultaneous sentence encoding, reducing computation time while maintaining high translation accuracy.

Findings

Achieves competitive accuracy on WMT'16 English-Romanian translation.

Outperforms recent results on WMT'15 English-German translation.

Speeds up CPU decoding by over two times without sacrificing accuracy.

Abstract

The prevalent approach to neural machine translation relies on bi-directional LSTMs to encode the source sentence. In this paper we present a faster and simpler architecture based on a succession of convolutional layers. This allows to encode the entire source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies. On WMT'16 English-Romanian translation we achieve competitive accuracy to the state-of-the-art and we outperform several recently published results on the WMT'15 English-German task. Our models obtain almost the same accuracy as a very deep LSTM setup on WMT'14 English-French translation. Our convolutional encoder speeds up CPU decoding by more than two times at the same or higher accuracy as a strong bi-directional LSTM baseline.