Enhancing Reinforcement Learning with discrete interfaces to learn the Dyck Language

TL;DR

This paper introduces a novel reinforcement learning approach that incorporates discrete interfaces to enable neural networks to understand and generate hierarchical structures like the Dyck language, achieving significant generalization and efficiency.

Contribution

It presents the first neural network solution that learns to utilize discrete data structures for hierarchical language understanding in reinforcement learning.

Findings

Model generalizes to sequences ten times longer than training data

Pre-training on execution traces improves training stability

Resulting model is small and fast

Abstract

Even though most interfaces in the real world are discrete, no efficient way exists to train neural networks to make use of them, yet. We enhance an Interaction Network (a Reinforcement Learning architecture) with discrete interfaces and train it on the generalized Dyck language. This task requires an understanding of hierarchical structures to solve, and has long proven difficult for neural networks. We provide the first solution based on learning to use discrete data structures. We encountered unexpected anomalous behavior during training, and utilized pre-training based on execution traces to overcome them. The resulting model is very small and fast, and generalizes to sequences that are an entire order of magnitude longer than the training data.