Learning Sentence Embeddings for Coherence Modelling and Beyond

TL;DR

This paper introduces a self-supervised sentence embedding technique that improves text coherence modeling, offers deeper data insights, and benefits writing and reading processes, achieving competitive performance with simpler methods.

Contribution

The authors develop a novel self-supervised sentence embedding method based on predicting sentence positions, enhancing coherence tasks and interpretability in NLP.

Findings

Embeddings achieve state-of-the-art performance on coherence tasks.

Embeddings provide useful insights for writers and readers.

Method outperforms complex approaches with simpler heuristics.

Abstract

We present a novel and effective technique for performing text coherence tasks while facilitating deeper insights into the data. Despite obtaining ever-increasing task performance, modern deep-learning approaches to NLP tasks often only provide users with the final network decision and no additional understanding of the data. In this work, we show that a new type of sentence embedding learned through self-supervision can be applied effectively to text coherence tasks while serving as a window through which deeper understanding of the data can be obtained. To produce these sentence embeddings, we train a recurrent neural network to take individual sentences and predict their location in a document in the form of a distribution over locations. We demonstrate that these embeddings, combined with simple visual heuristics, can be used to achieve performance competitive with state-of-the-art on multiple text coherence tasks, outperforming more complex and specialized approaches. Additionally, we demonstrate that these embeddings can provide insights useful to writers for improving writing quality and informing document structuring, and assisting readers in summarizing and locating information.