Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs

TL;DR

This paper evaluates the effectiveness of vision-capable large language models in automatically grading handwritten mathematical responses, highlighting their accuracy and common error modes.

Contribution

It extends existing LLM-based grading pipelines to handwritten math, providing an empirical assessment of accuracy and error sources in real educational settings.

Findings

High overall accuracy in grading handwritten math responses

87% accuracy with most errors due to transcription failures

Identified error modes include image quality issues and hallucinated content

Abstract

Automated grading systems have enabled scalable assessment for many response types, but handwritten mathematics remains a barrier due to the complexity of multi-step solutions. Vision-capable large language models (LLMs) offer new opportunities here, yet their reliability in authentic instructional settings remains poorly understood. We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics. Extending a prior pipeline for typed responses, we integrate transcription and rubric-based evaluation of photographic submissions within a single LLM call, evaluating on student work from two university STEM courses. Comparing AI grading decisions against human-assigned ground truth at the rubric-item level, we observe high overall accuracy, with most errors -- 87\% in the best model -- attributable to transcription failures rather than rubric misapplication. We categorize common error modes, including image quality issues, hallucinated content, and incorrect handling of equivalent expressions. These findings highlight both the promise and limitations of LLM-based grading for handwritten mathematics, providing guidance for system design, prompt refinement, and deployment in educational settings.