Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams

TL;DR

This study evaluates GPT-4o's ability to reliably score handwritten physics responses using skill-based rubrics, highlighting the importance of rubric clarity and LLM configuration for consistent AI-assisted grading.

Contribution

It demonstrates that well-structured rubrics and controlled LLM settings can achieve scoring reliability comparable to human raters in STEM assessments.

Findings

AI scoring agreement matches human inter-rater reliability for high- and low-performing responses.

Checklist-based rubrics improve scoring consistency over holistic approaches.

Prompting format influences AI scoring slightly, temperature has limited effect.

Abstract

Student responses in STEM assessments are often handwritten and combine symbolic expressions, calculations, and diagrams, creating substantial variation in format and interpretation. Despite their importance for evaluating students' reasoning, such responses are time-consuming to score and prone to rater inconsistency, particularly when partial credit is required. Recent advances in large language models (LLMs) have increased attention to AI-assisted scoring, yet evidence remains limited regarding how rubric design and LLM configurations influence reliability across performance levels. This study examined the reliability of AI-assisted scoring of undergraduate physics constructed responses using GPT-4o. Twenty authentic handwritten exam responses were scored across two rounds by four instructors and by the AI model using skill-based rubrics with differing levels of analytic granularity. Prompting format and temperature settings were systematically varied. Overall, human-AI agreement on total scores was comparable to human inter-rater reliability and was highest for high- and low-performing responses, but declined for mid-level responses involving partial or ambiguous reasoning. Criterion-level analyses showed stronger alignment for clearly defined conceptual skills than for extended procedural judgments. A more fine-grained, checklist-based rubric improved consistency relative to holistic scoring. These findings indicate that reliable AI-assisted scoring depends primarily on clear, well-structured rubrics, while prompting format plays a secondary role and temperature has relatively limited impact. More broadly, the study provides transferable design recommendations for implementing reliable LLM-assisted scoring in STEM contexts through skill-based rubrics and controlled LLM settings.