Document Context Language Models

TL;DR

This paper introduces Document-Context Language Models (DCLM), a new neural network approach that incorporates multi-level discourse information to improve document coherence and predictive performance.

Contribution

The paper presents a novel multi-level recurrent neural network model that effectively integrates discourse structure for improved language modeling.

Findings

DCLM models achieve better predictive likelihoods than word-level models.

DCLM models significantly improve assessments of document coherence.

Empirical evaluation demonstrates the effectiveness of incorporating discourse structure.

Abstract

Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.