Grading Assistance for a Handwritten Thermodynamics Exam using Artificial Intelligence: An Exploratory Study

TL;DR

This study explores AI-assisted grading of handwritten thermodynamics exams, highlighting challenges in digitizing handwritten answers, grading granularity, and graphic evaluation, with insights into system reliability and areas needing human oversight.

Contribution

It evaluates four AI workflows for grading handwritten solutions, identifying key challenges and providing recommendations to improve automated assessment accuracy.

Findings

Digitizing handwritten answers remains a major challenge.

Fine-grained rubrics increase grading errors.

Graphics are harder to evaluate than mathematical derivations.

Abstract

Using a high-stakes thermodynamics exam as sample (252~students, four multipart problems), we investigate the viability of four workflows for AI-assisted grading of handwritten student solutions. We find that the greatest challenge lies in converting handwritten answers into a machine-readable format. The granularity of grading criteria also influences grading performance: employing a fine-grained rubric for entire problems often leads to bookkeeping errors and grading failures, while grading problems in parts is more reliable but tends to miss nuances. We also found that grading hand-drawn graphics, such as process diagrams, is less reliable than mathematical derivations due to the difficulty in differentiating essential details from extraneous information. Although the system is precise in identifying exams that meet passing criteria, exams with failing grades still require human grading. We conclude with recommendations to overcome some of the encountered challenges.