Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

TL;DR

This paper explores how stack-augmented recurrent neural networks can learn algorithmic patterns, such as counting and memorization, which standard models struggle with, advancing the capacity of neural networks for complex sequence prediction.

Contribution

It introduces a novel stack-augmented recurrent network that can learn basic algorithms from sequential data, overcoming limitations of standard recurrent models.

Findings

Stack-augmented RNNs can learn counting and memorization tasks.

Standard RNNs fail on certain algorithmic sequence prediction tasks.

The proposed model extends the capabilities of neural networks for structured sequence learning.

Abstract

Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.