Context is Key: New Approaches to Neural Coherence Modeling

TL;DR

This paper introduces two novel methods for neural coherence modeling, framing it as a regression task, and demonstrates that simpler models can match or outperform complex approaches in coherence evaluation metrics.

Contribution

The paper proposes two new methods—'first-next' and context concatenation—for neural coherence modeling, improving efficiency and performance over existing techniques.

Findings

Achieved state-of-the-art Kendall-tau distance scores.

Matched or exceeded current metrics in positional accuracy.

Simpler models can replicate complex approach gains.

Abstract

We formulate coherence modeling as a regression task and propose two novel methods to combine techniques from our setup with pairwise approaches. The first of our methods is a model that we call "first-next," which operates similarly to selection sorting but conditions decision-making on information about already-sorted sentences. The second consists of a technique for adding context to regression-based models by concatenating sentence-level representations with an encoding of its corresponding out-of-order paragraph. This latter model achieves Kendall-tau distance and positional accuracy scores that match or exceed the current state-of-the-art on these metrics. Our results suggest that many of the gains that come from more complex, machine-translation inspired approaches can be achieved with simpler, more efficient models.