Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach

TL;DR

This paper introduces a multi-task neural language model that jointly predicts words and syntactic distances, leveraging syntactic structure to improve language modeling performance and parse tree quality.

Contribution

It proposes a novel multi-task training approach that incorporates syntactic distances into neural language models for enhanced performance.

Findings

Lower perplexity on Penn Treebank and Chinese Treebank datasets

Improved quality of induced parse trees

Effective integration of syntactic structure into language modeling

Abstract

It is commonly believed that knowledge of syntactic structure should improve language modeling. However, effectively and computationally efficiently incorporating syntactic structure into neural language models has been a challenging topic. In this paper, we make use of a multi-task objective, i.e., the models simultaneously predict words as well as ground truth parse trees in a form called "syntactic distances", where information between these two separate objectives shares the same intermediate representation. Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training signals, the model is able to achieve lower perplexity and induce trees with better quality.