Exploring LLM Autoscoring Reliability in Large-Scale Writing Assessments Using Generalizability Theory

TL;DR

This study assesses the reliability of large language models in scoring AP Chinese writing tasks, comparing their consistency to human raters using generalizability theory, and explores hybrid scoring approaches for improved reliability.

Contribution

It introduces a novel application of generalizability theory to evaluate LLM scoring reliability in large-scale assessments and compares AI and human rater consistency.

Findings

LLMs show reasonable scoring consistency for story narration tasks.

Hybrid scoring models combining human and AI raters improve reliability.

Human raters produce more reliable scores overall.

Abstract

This study investigates the estimation of reliability for large language models (LLMs) in scoring writing tasks from the AP Chinese Language and Culture Exam. Using generalizability theory, the research evaluates and compares score consistency between human and AI raters across two types of AP Chinese free-response writing tasks: story narration and email response. These essays were independently scored by two trained human raters and seven AI raters. Each essay received four scores: one holistic score and three analytic scores corresponding to the domains of task completion, delivery, and language use. Results indicate that although human raters produced more reliable scores overall, LLMs demonstrated reasonable consistency under certain conditions, particularly for story narration tasks. Composite scoring that incorporates both human and AI raters improved reliability, which supports that hybrid scoring models may offer benefits for large-scale writing assessments.