Learning Context-Free Languages with Nondeterministic Stack RNNs

TL;DR

This paper introduces a differentiable nondeterministic stack data structure integrated with RNNs, enabling better learning of context-free languages by efficiently representing multiple configurations and outperforming existing models on formal language tasks.

Contribution

The paper proposes a novel nondeterministic stack RNN that simulates nondeterministic pushdown automata, improving learning and generalization on context-free language tasks.

Findings

More reliable convergence to algorithmic behavior on deterministic tasks

Lower cross-entropy on nondeterministic tasks

Outperforms existing stack RNNs in formal language benchmarks

Abstract

We present a differentiable stack data structure that simultaneously and tractably encodes an exponential number of stack configurations, based on Lang's algorithm for simulating nondeterministic pushdown automata. We call the combination of this data structure with a recurrent neural network (RNN) controller a Nondeterministic Stack RNN. We compare our model against existing stack RNNs on various formal languages, demonstrating that our model converges more reliably to algorithmic behavior on deterministic tasks, and achieves lower cross-entropy on inherently nondeterministic tasks.