Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays

TL;DR

This paper explores trait-based automatic scoring of argumentative essays using structured in-context learning with small open-source LLMs and a supervised BigBird model with ordinal regression, improving interpretability and alignment with educational rubrics.

Contribution

It introduces a dual modeling approach combining LLM prompting and an ordinal regression model to enhance trait-level essay scoring accuracy and interpretability.

Findings

Explicit ordinal modeling improves agreement with human raters.

Small open-source LLMs perform competitively without fine-tuning.

Trait-based scoring offers more pedagogical insights than holistic scores.

Abstract

Automated Essay Scoring systems have traditionally focused on holistic scores, limiting their pedagogical usefulness, especially in the case of complex essay genres such as argumentative writing. In educational contexts, teachers and learners require interpretable, trait-level feedback that aligns with instructional goals and established rubrics. In this paper, we study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms designed for realistic educational deployment: (1) structured in-context learning with small open-source LLMs, and (2) a supervised, encoder-based BigBird model with a CORAL-style ordinal regression formulation, optimized for long-sequence understanding. We conduct a systematic evaluation on the ASAP++ dataset, which includes essay scores across five quality traits, offering strong coverage of core argumentation dimensions. LLMs are prompted with designed, rubric-aligned in-context examples, along with feedback and confidence requests, while we explicitly model ordinality in scores with the BigBird model via the rank-consistent CORAL framework. Our results show that explicitly modeling score ordinality substantially improves agreement with human raters across all traits, outperforming LLMs and nominal classification and regression-based baselines. This finding reinforces the importance of aligning model objectives with rubric semantics for educational assessment. At the same time, small open-source LLMs achieve a competitive performance without task-specific fine-tuning, particularly for reasoning-oriented traits, while enabling transparent, privacy-preserving, and locally deployable assessment scenarios. Our findings provide methodological, modeling, and practical insights for the design of AI-based educational systems that aim to deliver interpretable, rubric-aligned feedback for argumentative writing.