# Finding Syntactic Representations in Neural Stacks

## TL;DR

This paper investigates whether neural stack models learn hierarchical syntactic structures by extracting constituency trees from their pushing behavior, showing they do infer linguistically relevant hierarchies.

## Contribution

It demonstrates that stack RNNs trained on language tasks can implicitly learn and produce syntactic constituency structures.

## Key findings

- Models produce parses reflecting natural language syntax
- Stack RNNs infer linguistically relevant hierarchical structures
- Unsupervised extraction reveals syntactic representations

## Abstract

Neural network architectures have been augmented with differentiable stacks in order to introduce a bias toward learning hierarchy-sensitive regularities. It has, however, proven difficult to assess the degree to which such a bias is effective, as the operation of the differentiable stack is not always interpretable. In this paper, we attempt to detect the presence of latent representations of hierarchical structure through an exploration of the unsupervised learning of constituency structure. Using a technique due to Shen et al. (2018a,b), we extract syntactic trees from the pushing behavior of stack RNNs trained on language modeling and classification objectives. We find that our models produce parses that reflect natural language syntactic constituencies, demonstrating that stack RNNs do indeed infer linguistically relevant hierarchical structure.