LSTMs Compose (and Learn) Bottom-Up

TL;DR

This paper investigates how LSTM language models develop hierarchical, compositional representations during training, revealing that they learn structures bottom-up by building on shorter constituents rather than from long-range relations.

Contribution

It introduces a measure of Decompositional Interdependence to analyze LSTM representations and demonstrates that hierarchical structures are learned bottom-up during training.

Findings

DI is higher on word pairs with lower syntactic distance

LSTM representations are learned bottom-up, relying on shorter constituents

Hierarchical structures emerge during training through bottom-up composition

Abstract

Recent work in NLP shows that LSTM language models capture hierarchical structure in language data. In contrast to existing work, we consider the \textit{learning} process that leads to their compositional behavior. For a closer look at how an LSTM's sequential representations are composed hierarchically, we present a related measure of Decompositional Interdependence (DI) between word meanings in an LSTM, based on their gate interactions. We connect this measure to syntax with experiments on English language data, where DI is higher on pairs of words with lower syntactic distance. To explore the inductive biases that cause these compositional representations to arise during training, we conduct simple experiments on synthetic data. These synthetic experiments support a specific hypothesis about how hierarchical structures are discovered over the course of training: that LSTM constituent representations are learned bottom-up, relying on effective representations of their shorter children, rather than learning the longer-range relations independently from children.