Context is Enough: Empirical Validation of $\textit{Sequentiality}$ on Essays

TL;DR

This paper empirically validates that a context-based measure of narrative flow, called sequentiality, aligns better with human judgments of essay quality than topic-based measures, and enhances automated scoring when combined with linguistic features.

Contribution

The study confirms that a context-only sequentiality metric is more valid and interpretable than topic-based versions, improving automated essay scoring models.

Findings

Contextual sequentiality correlates more strongly with human judgments.

Combining contextual sequentiality with linguistic features improves prediction accuracy.

Context-based sequentiality outperforms original and topic-only measures in validation.

Abstract

Recent work has proposed using Large Language Models (LLMs) to quantify narrative flow through a measure called sequentiality, which combines topic and contextual terms. A recent critique argued that the original results were confounded by how topics were selected for the topic-based component, and noted that the metric had not been validated against ground-truth measures of flow. That work proposed using only the contextual term as a more conceptually valid and interpretable alternative. In this paper, we empirically validate that proposal. Using two essay datasets with human-annotated trait scores, ASAP++ and ELLIPSE, we show that the contextual version of sequentiality aligns more closely with human assessments of discourse-level traits such as Organization and Cohesion. While zero-shot prompted LLMs predict trait scores more accurately than the contextual measure alone, the contextual measure adds more predictive value than both the topic-only and original sequentiality formulations when combined with standard linguistic features. Notably, this combination also outperforms the zero-shot LLM predictions, highlighting the value of explicitly modeling sentence-to-sentence flow. Our findings support the use of context-based sequentiality as a validated, interpretable, and complementary feature for automated essay scoring and related NLP tasks.