Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards

TL;DR

This paper introduces SaMRL, a reinforcement learning approach for multi-trait automated essay scoring that directly incorporates evaluation metrics into training, improving score prediction accuracy and robustness.

Contribution

It proposes a novel reinforcement learning framework with scoring-aware rewards for multi-trait AES, enabling direct optimization of agreement metrics like QWK.

Findings

SaMRL improves scoring accuracy on challenging prompts.

The autoregressive framework leverages token probabilities for robust predictions.

Empirical results show enhanced model training and performance.

Abstract

Recent advances in automated essay scoring (AES) have shifted towards evaluating multiple traits to provide enriched feedback. Like typical AES systems, multi-trait AES employs the quadratic weighted kappa (QWK) to measure agreement with human raters, aligning closely with the rating schema; however, its non-differentiable nature prevents its direct use in neural network training. In this paper, we propose Scoring-aware Multi-reward Reinforcement Learning (SaMRL), which integrates actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation, as RL requires probability distributions; instead, we adopt an autoregressive score generation framework to leverage token generation probabilities for robust multi-trait score predictions. Empirical analyses demonstrate that SaMRL facilitates model training, notably enhancing scoring of previously inferior prompts.