Context-Free Transductions with Neural Stacks

TL;DR

This paper investigates stack-augmented RNNs, demonstrating their ability to learn intuitive stack-based strategies for formal language tasks, while also highlighting training difficulties and alternative unstructured memory usage.

Contribution

It provides an analysis of stack RNN behavior on formal language tasks, revealing their capacity for stack-based strategies and comparing their training challenges to classical models.

Findings

Stack RNNs can learn stack-based strategies for tasks like string reversal.

Training stack RNNs is more difficult than training LSTMs.

Complex networks often use the stack as unstructured memory rather than for stack strategies.

Abstract

This paper analyzes the behavior of stack-augmented recurrent neural network (RNN) models. Due to the architectural similarity between stack RNNs and pushdown transducers, we train stack RNN models on a number of tasks, including string reversal, context-free language modelling, and cumulative XOR evaluation. Examining the behavior of our networks, we show that stack-augmented RNNs can discover intuitive stack-based strategies for solving our tasks. However, stack RNNs are more difficult to train than classical architectures such as LSTMs. Rather than employ stack-based strategies, more complex networks often find approximate solutions by using the stack as unstructured memory.