Sentence Ordering and Coherence Modeling using Recurrent Neural Networks

TL;DR

This paper introduces an unsupervised deep learning model using recurrent neural networks to improve sentence ordering and coherence modeling, demonstrating superior performance and meaningful sentence representations that capture logical paragraph structure.

Contribution

The paper presents a novel set-to-sequence RNN framework for sentence ordering, outperforming previous methods and producing high-quality representations for various NLP tasks.

Findings

Outperforms prior methods in order discrimination and abstract ordering tasks

Learns sentence representations capturing high-level logical structure

Comparable performance to state-of-the-art pre-training methods on similarity tasks

Abstract

Modeling the structure of coherent texts is a key NLP problem. The task of coherently organizing a given set of sentences has been commonly used to build and evaluate models that understand such structure. We propose an end-to-end unsupervised deep learning approach based on the set-to-sequence framework to address this problem. Our model strongly outperforms prior methods in the order discrimination task and a novel task of ordering abstracts from scientific articles. Furthermore, our work shows that useful text representations can be obtained by learning to order sentences. Visualizing the learned sentence representations shows that the model captures high-level logical structure in paragraphs. Our representations perform comparably to state-of-the-art pre-training methods on sentence similarity and paraphrase detection tasks.