Mark My Works Autograder for Programming Courses

TL;DR

Mark My Works is an autograding system that combines unit testing with large language model explanations to provide detailed, pedagogical feedback on student programming submissions, aiming to improve feedback quality in large courses.

Contribution

The paper introduces a novel autograder integrating LLM-generated explanations with traditional testing, enhancing feedback detail and transparency in programming education.

Findings

AI scores did not significantly correlate with human scores (r = -0.177, p = 0.124).

Both AI and human grading showed similar quality hierarchies despite different scoring philosophies.

AI provided more detailed technical feedback and scored more conservatively.

Abstract

Large programming courses struggle to provide timely, detailed feedback on student code. We developed Mark My Works, a local autograding system that combines traditional unit testing with LLM-generated explanations. The system uses role-based prompts to analyze submissions, critique code quality, and generate pedagogical feedback while maintaining transparency in its reasoning process. We piloted the system in a 191-student engineering course, comparing AI-generated assessments with human grading on 79 submissions. While AI scores showed no linear correlation with human scores (r = -0.177, p = 0.124), both systems exhibited similar left-skewed distributions, suggesting they recognize comparable quality hierarchies despite different scoring philosophies. The AI system demonstrated more conservative scoring (mean: 59.95 vs 80.53 human) but generated significantly more detailed technical feedback.