Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays
Lucile Favero, Juan Antonio P\'erez-Ortiz, Tanja K\"aser, Nuria Oliver

TL;DR
This paper explores trait-based automatic scoring of argumentative essays using structured in-context learning with small open-source LLMs and a supervised BigBird model with ordinal regression, improving interpretability and alignment with educational rubrics.
Contribution
It introduces a dual modeling approach combining LLM prompting and an ordinal regression model to enhance trait-level essay scoring accuracy and interpretability.
Findings
Explicit ordinal modeling improves agreement with human raters.
Small open-source LLMs perform competitively without fine-tuning.
Trait-based scoring offers more pedagogical insights than holistic scores.
Abstract
Automated Essay Scoring systems have traditionally focused on holistic scores, limiting their pedagogical usefulness, especially in the case of complex essay genres such as argumentative writing. In educational contexts, teachers and learners require interpretable, trait-level feedback that aligns with instructional goals and established rubrics. In this paper, we study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms designed for realistic educational deployment: (1) structured in-context learning with small open-source LLMs, and (2) a supervised, encoder-based BigBird model with a CORAL-style ordinal regression formulation, optimized for long-sequence understanding. We conduct a systematic evaluation on the ASAP++ dataset, which includes essay scores across five quality traits, offering strong coverage of core argumentation dimensions. LLMs…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIntelligent Tutoring Systems and Adaptive Learning · Topic Modeling · Innovative Teaching and Learning Methods
