Grammar Induction with Neural Language Models: An Unusual Replication

TL;DR

This paper critically examines and replicates recent neural latent tree learning models, confirming their robustness and suggesting neural language models are promising for grammar induction.

Contribution

It provides a fair replication of previous results, extends the analysis to new datasets, and establishes neural language models as a viable approach for grammar induction.

Findings

All model variants outperform latent tree learning baselines.

Models perform competitively with symbolic grammar induction systems.

Neural language modeling shows empirical success for grammar induction.

Abstract

A substantial thread of recent work on latent tree learning has attempted to develop neural network models with parse-valued latent variables and train them on non-parsing tasks, in the hope of having them discover interpretable tree structure. In a recent paper, Shen et al. (2018) introduce such a model and report near-state-of-the-art results on the target task of language modeling, and the first strong latent tree learning result on constituency parsing. In an attempt to reproduce these results, we discover issues that make the original results hard to trust, including tuning and even training on what is effectively the test set. Here, we attempt to reproduce these results in a fair experiment and to extend them to two new datasets. We find that the results of this work are robust: All variants of the model under study outperform all latent tree learning baselines, and perform competitively with symbolic grammar induction systems. We find that this model represents the first empirical success for latent tree learning, and that neural network language modeling warrants further study as a setting for grammar induction.