Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test

TL;DR

This paper critically examines the Shuffle Test for coherence measurement in NLP, demonstrating that models can achieve high accuracy through simple finetuning, and proposes a new, more challenging variant to better evaluate true coherence understanding.

Contribution

The paper advocates for zero-shot evaluation of coherence models and introduces the k-Block Shuffle Test to better assess models' genuine understanding of text coherence.

Findings

Finetuned RoBERTa achieves 97.8% accuracy on the Shuffle Test.

Larger models perform well out-of-the-box in zero-shot settings.

The k-Block Shuffle Test reduces model performance, highlighting its effectiveness as a benchmark.

Abstract

The Shuffle Test is the most common task to evaluate whether NLP models can measure coherence in text. Most recent work uses direct supervision on the task; we show that by simply finetuning a RoBERTa model, we can achieve a near perfect accuracy of 97.8%, a state-of-the-art. We argue that this outstanding performance is unlikely to lead to a good model of text coherence, and suggest that the Shuffle Test should be approached in a Zero-Shot setting: models should be evaluated without being trained on the task itself. We evaluate common models in this setting, such as Generative and Bi-directional Transformers, and find that larger architectures achieve high-performance out-of-the-box. Finally, we suggest the k-Block Shuffle Test, a modification of the original by increasing the size of blocks shuffled. Even though human reader performance remains high (around 95% accuracy), model performance drops from 94% to 78% as block size increases, creating a conceptually simple challenge to benchmark NLP models. Code available: https://github.com/tingofurro/shuffle_test/