TRATES: Trait-Specific Rubric-Assisted Cross-Prompt Essay Scoring

TL;DR

TRATES introduces a novel framework for trait-specific automated essay scoring that leverages large language models and rubrics to generate features, achieving state-of-the-art results across multiple traits.

Contribution

It presents a new trait-specific, rubric-based AES framework using LLMs to generate assessment features, improving scoring accuracy across prompts.

Findings

Achieves state-of-the-art performance on a widely-used dataset.

LLM-generated features are the most significant contributors.

Effective across multiple traits and prompts.

Abstract

Research on holistic Automated Essay Scoring (AES) is long-dated; yet, there is a notable lack of attention for assessing essays according to individual traits. In this work, we propose TRATES, a novel trait-specific and rubric-based cross-prompt AES framework that is generic yet specific to the underlying trait. The framework leverages a Large Language Model (LLM) that utilizes the trait grading rubrics to generate trait-specific features (represented by assessment questions), then assesses those features given an essay. The trait-specific features are eventually combined with generic writing-quality and prompt-specific features to train a simple classical regression model that predicts trait scores of essays from an unseen prompt. Experiments show that TRATES achieves a new state-of-the-art performance across all traits on a widely-used dataset, with the generated LLM-based features being the most significant.