Linguistic Versus Latent Relations for Modeling Coherent Flow in Paragraphs

TL;DR

This paper compares linguistic and latent relation models for generating coherent paragraphs, demonstrating that incorporating both types of relations improves paragraph coherence in language models.

Contribution

It introduces two novel models that integrate linguistic and latent relations into paragraph generation, enhancing coherence over previous approaches.

Findings

Both models outperform baselines in paragraph generation tasks.

Supervised model effectively learns discourse structures.

Unsupervised model captures latent sentence relations.

Abstract

Generating a long, coherent text such as a paragraph requires a high-level control of different levels of relations between sentences (e.g., tense, coreference). We call such a logical connection between sentences as a (paragraph) flow. In order to produce a coherent flow of text, we explore two forms of intersentential relations in a paragraph: one is a human-created linguistical relation that forms a structure (e.g., discourse tree) and the other is a relation from latent representation learned from the sentences themselves. Our two proposed models incorporate each form of relations into document-level language models: the former is a supervised model that jointly learns a language model as well as discourse relation prediction, and the latter is an unsupervised model that is hierarchically conditioned by a recurrent neural network (RNN) over the latent information. Our proposed models with both forms of relations outperform the baselines in partially conditioned paragraph generation task. Our codes and data are publicly available.