Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLP

TL;DR

This study demonstrates that large language models can reliably automate scoring of nationwide school-leaving essays, matching human performance and enabling scalable, fair, and detailed feedback within a principled, rubric-driven framework.

Contribution

It introduces a rubric-based, human-in-the-loop scoring pipeline using LLMs for high-stakes national assessments, validated on Estonian exam data.

Findings

Automated scoring achieves performance comparable to human raters.

The system provides detailed, personalized feedback for students.

The approach is viable for small-language contexts and large-scale implementation.

Abstract

Large language models (LLMs) enable rapid and consistent automated evaluation of open-ended exam responses, including dimensions of content and argumentation that have traditionally required human judgment. This is particularly important in cases where a large amount of exams need to be graded in a limited time frame, such as nation-wide graduation exams in various countries. Here, we examine the applicability of automated scoring on two large datasets of trial exam essays of two full national cohorts from Estonia. We operationalize the official curriculum-based rubric and compare LLM and statistical natural language processing (NLP) based assessments with human panel scores. The results show that automated scoring can achieve performance comparable to that of human raters and tends to fall within the human scoring range. We also evaluate bias, prompt injection risks, and LLMs as essay writers. These findings demonstrate that a principled, rubric-driven, human-in-the-loop scoring pipeline is viable for high-stakes writing assessment, particularly relevant for digitally advanced societies like Estonia, which is about to adapt a fully electronic examination system. Furthermore, the system produces fine-grained subscore profiles that can be used to generate systematic, personalized feedback for instruction and exam preparation. The study provides evidence that LLM-assisted assessment can be implemented at a national scale, even in a small-language context, while maintaining human oversight and compliance with emerging educational and regulatory standards.