Neural Language Modeling by Jointly Learning Syntax and Lexicon

TL;DR

This paper introduces PRPN, a neural language model that jointly induces syntactic structures from unannotated text and improves language modeling performance by leveraging these structures, all trained via backpropagation.

Contribution

The paper presents a novel neural model that simultaneously learns syntax and language modeling without supervision, outperforming previous methods.

Findings

Successfully induces syntactic structures from unannotated data

Achieves state-of-the-art results on language modeling tasks

Enables end-to-end training with backpropagation through structure induction

Abstract

We propose a neural language model capable of unsupervised syntactic structure induction. The model leverages the structure information to form better semantic representations and better language modeling. Standard recurrent neural networks are limited by their structure and fail to efficiently use syntactic information. On the other hand, tree-structured recursive networks usually require additional structural supervision at the cost of human expert annotation. In this paper, We propose a novel neural language model, called the Parsing-Reading-Predict Networks (PRPN), that can simultaneously induce the syntactic structure from unannotated sentences and leverage the inferred structure to learn a better language model. In our model, the gradient can be directly back-propagated from the language model loss into the neural parsing network. Experiments show that the proposed model can discover the underlying syntactic structure and achieve state-of-the-art performance on word/character-level language model tasks.