Tree-structured composition in neural networks without tree-structured architectures

TL;DR

This paper investigates whether sequence models like LSTMs can implicitly learn recursive tree structures in language tasks, finding they can but are less effective than explicit tree-structured models.

Contribution

The study demonstrates that LSTMs can discover recursive structures in data, but explicit tree-structured models outperform them in exploiting such structures.

Findings

LSTMs can learn to utilize recursive structure in artificial tasks.

Tree-structured models outperform LSTMs in recursive structure tasks.

LSTMs lag behind tree models even with large training data.

Abstract

Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying tree structure. However, its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.