Is GPT-4 Alone Sufficient for Automated Essay Scoring?: A Comparative Judgment Approach Based on Rater Cognition

TL;DR

This paper introduces a novel approach combining Large Language Models and Comparative Judgment to improve automated essay scoring, outperforming traditional rubric-based methods without the need for fine-tuning.

Contribution

It proposes a zero-shot comparative judgment method leveraging LLMs for AES, addressing the limitations of existing scoring approaches.

Findings

CJ method outperforms traditional rubric-based scoring

Zero-shot prompting effectively compares essays

Approach reduces need for task-specific fine-tuning

Abstract

Large Language Models (LLMs) have shown promise in Automated Essay Scoring (AES), but their zero-shot and few-shot performance often falls short compared to state-of-the-art models and human raters. However, fine-tuning LLMs for each specific task is impractical due to the variety of essay prompts and rubrics used in real-world educational contexts. This study proposes a novel approach combining LLMs and Comparative Judgment (CJ) for AES, using zero-shot prompting to choose between two essays. We demonstrate that a CJ method surpasses traditional rubric-based scoring in essay scoring using LLMs.