A Deep Memory-based Architecture for Sequence-to-Sequence Learning

TL;DR

DEEPMEMORY is a deep neural architecture for sequence-to-sequence tasks like translation, utilizing stacked memory layers and learned nonlinear transformations to improve over existing models and achieve competitive performance.

Contribution

It introduces a deep memory-based architecture inspired by Neural Turing Machines, enabling more complex sequence modeling for translation tasks.

Findings

Outperforms previous neural translation models with a deeper architecture.

Achieves performance comparable to traditional phrase-based systems.

Easily scalable to large datasets.

Abstract

We propose DEEPMEMORY, a novel deep architecture for sequence-to-sequence learning, which performs the task through a series of nonlinear transformations from the representation of the input sequence (e.g., a Chinese sentence) to the final output sequence (e.g., translation to English). Inspired by the recently proposed Neural Turing Machine (Graves et al., 2014), we store the intermediate representations in stacked layers of memories, and use read-write operations on the memories to realize the nonlinear transformations between the representations. The types of transformations are designed in advance but the parameters are learned from data. Through layer-by-layer transformations, DEEPMEMORY can model complicated relations between sequences necessary for applications such as machine translation between distant languages. The architecture can be trained with normal back-propagation on sequenceto-sequence data, and the learning can be easily scaled up to a large corpus. DEEPMEMORY is broad enough to subsume the state-of-the-art neural translation model in (Bahdanau et al., 2015) as its special case, while significantly improving upon the model with its deeper architecture. Remarkably, DEEPMEMORY, being purely neural network-based, can achieve performance comparable to the traditional phrase-based machine translation system Moses with a small vocabulary and a modest parameter size.