Analyzing Neural Discourse Coherence Models

TL;DR

This paper systematically evaluates current neural discourse coherence models by testing their sensitivity to linguistic alterations and analyzing their encoded knowledge, aiming to improve coherence assessment methods.

Contribution

It introduces new datasets with linguistic alterations and provides insights into model sensitivities and encoded discourse knowledge.

Findings

Models show varying sensitivity to syntactic and semantic changes.

Discourse embeddings encode specific coherence-related information.

Public datasets enable standardized testing of coherence models.

Abstract

In this work, we systematically investigate how well current models of coherence can capture aspects of text implicated in discourse organisation. We devise two datasets of various linguistic alterations that undermine coherence and test model sensitivity to changes in syntax and semantics. We furthermore probe discourse embedding space and examine the knowledge that is encoded in representations of coherence. We hope this study shall provide further insight into how to frame the task and improve models of coherence assessment further. Finally, we make our datasets publicly available as a resource for researchers to use to test discourse coherence models.