Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages

TL;DR

This paper demonstrates that memory-augmented recurrent neural networks can learn complex hierarchical languages like generalized Dyck languages, surpassing simple RNNs in modeling capacity and emulating pushdown automata.

Contribution

Introduction of three memory-augmented RNN architectures capable of recognizing hierarchical languages, including generalized Dyck languages, with end-to-end training.

Findings

Memory-augmented RNNs recognize Dyck languages with up to six parenthesis pairs.

Models successfully learn palindrome and string-reversal tasks.

Memory augmentation enhances the modeling capacity over simple RNNs.

Abstract

We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and explore their capabilities on a series of simple language modeling tasks whose solutions require stack-based mechanisms. We provide the first demonstration of neural networks recognizing the generalized Dyck languages, which express the core of what it means to be a language with hierarchical structure. Our memory-augmented architectures are easy to train in an end-to-end fashion and can learn the Dyck languages over as many as six parenthesis-pairs, in addition to two deterministic palindrome languages and the string-reversal transduction task, by emulating pushdown automata. Our experiments highlight the increased modeling capacity of memory-augmented models over simple RNNs, while inflecting our understanding of the limitations of these models.