CodEv: An Automated Grading Framework Leveraging Large Language Models for Consistent and Constructive Feedback

TL;DR

CodEv is an automated grading framework that uses Large Language Models with Chain of Thought prompting and ensembles to provide consistent, accurate, and human-aligned feedback on programming assignments.

Contribution

This paper introduces CodEv, a novel framework combining LLMs, Chain of Thought prompting, and ensemble methods for reliable automated programming assignment grading.

Findings

Achieves grading accuracy comparable to human evaluators

Provides consistent and constructive feedback using smaller LLMs

Validated through extensive evaluation and agreement tests

Abstract

Grading programming assignments is crucial for guiding students to improve their programming skills and coding styles. This study presents an automated grading framework, CodEv, which leverages Large Language Models (LLMs) to provide consistent and constructive feedback. We incorporate Chain of Thought (CoT) prompting techniques to enhance the reasoning capabilities of LLMs and ensure that the grading is aligned with human evaluation. Our framework also integrates LLM ensembles to improve the accuracy and consistency of scores, along with agreement tests to deliver reliable feedback and code review comments. The results demonstrate that the framework can yield grading results comparable to human evaluators, by using smaller LLMs. Evaluation and consistency tests of the LLMs further validate our approach, confirming the reliability of the generated scores and feedback.