Learning to Transduce with Unbounded Memory

TL;DR

This paper introduces memory-augmented recurrent neural networks with differentiable data structures, demonstrating improved generalization and the ability to learn underlying algorithms in language transduction tasks.

Contribution

It proposes new neural architectures with differentiable stacks, queues, and deques, enhancing the representational power and generalization of recurrent models for transduction.

Findings

Memory-augmented networks outperform standard RNNs in experiments.

Differentiable data structures enable learning of underlying algorithms.

Models generalize well to synthetic and real transduction tasks.

Abstract

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.