Evaluating Generative AI for CS1 Code Grading: Direct vs Reverse Methods

TL;DR

This paper compares two AI-based methods for grading introductory programming assignments, evaluating their accuracy, efficiency, and consistency against human graders, and explores their potential for scalable, fair assessment.

Contribution

It introduces a novel 'Reverse' grading approach using LLMs, compares it with the traditional 'Direct' method, and assesses their effectiveness in automated CS1 code grading.

Findings

Reverse method offers more fine-grained assessment.

Direct method is faster and simpler to implement.

Both methods benefit from careful prompt engineering.

Abstract

Manual grading of programming assignments in introductory computer science courses can be time-consuming and prone to inconsistencies. While unit testing is commonly used for automatic evaluation, it typically follows a binary pass/fail model and does not give partial marks. Recent advances in large language models (LLMs) offer the potential for automated, scalable, and more objective grading. This paper compares two AI-based grading techniques: \textit{Direct}, where the AI model applies a rubric directly to student code, and \textit{Reverse} (a newly proposed approach), where the AI first fixes errors, then deduces a grade based on the nature and number of fixes. Each method was evaluated on both the instructor's original grading scale and a tenfold expanded scale to assess the impact of range on AI grading accuracy. To assess their effectiveness, AI-assigned scores were evaluated against human tutor evaluations on a range of coding problems and error types. Initial findings suggest that while the Direct approach is faster and straightforward, the Reverse technique often provides a more fine-grained assessment by focusing on correction effort. Both methods require careful prompt engineering, particularly for allocating partial credit and handling logic errors. To further test consistency, we also used synthetic student code generated using Gemini Flash 2.0, which allowed us to evaluate AI graders on a wider range of controlled error types and difficulty levels. We discuss the strengths and limitations of each approach, practical considerations for prompt design, and future directions for hybrid human-AI grading systems that aim to improve consistency, efficiency, and fairness in CS courses.