Large Language Models in Student Assessment: Comparing ChatGPT and Human Graders

TL;DR

This study evaluates GPT-4's effectiveness in grading master-level student essays, revealing it matches average scores but has limitations in reliability and sensitivity to nuanced grading criteria, highlighting AI's current potential and challenges in education.

Contribution

It provides a comparative analysis of GPT-4 and human graders in higher education assessment, emphasizing the model's current limitations and stability across different prompts.

Findings

GPT-4 aligns with human grades on average

GPT-4 shows low interrater reliability with humans

Prompt engineering does not significantly change GPT-4's grading performance

Abstract

This study investigates the efficacy of large language models (LLMs) as tools for grading master-level student essays. Utilizing a sample of 60 essays in political science, the study compares the accuracy of grades suggested by the GPT-4 model with those awarded by university teachers. Results indicate that while GPT-4 aligns with human grading standards on mean scores, it exhibits a risk-averse grading pattern and its interrater reliability with human raters is low. Furthermore, modifications in the grading instructions (prompt engineering) do not significantly alter AI performance, suggesting that GPT-4 primarily assesses generic essay characteristics such as language quality rather than adapting to nuanced grading criteria. These findings contribute to the understanding of AI's potential and limitations in higher education, highlighting the need for further development to enhance its adaptability and sensitivity to specific educational assessment requirements.