Benchmarking Educational Program Repair
Charles Koutcheme, Nicola Dainese, Sami Sarsa, Juho Leinonen, Arto, Hellas, Paul Denny

TL;DR
This paper introduces a standardized benchmark for educational program repair using large language models, including curated datasets, a new evaluation metric, and baseline model performance, to enable fair comparison of approaches.
Contribution
It presents a novel benchmark with curated datasets, a unified evaluation procedure, and a new metric for assessing program repair quality in educational contexts.
Findings
Baseline models show varying repair quality
The rouge@k metric correlates with repair effectiveness
Standardized evaluation enables fair comparison
Abstract
The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks. For example, recent work in programming education has used LLMs to generate learning resources, improve error messages, and provide feedback on code. However, one factor that limits progress within the field is that much of the research uses bespoke datasets and different evaluation metrics, making direct comparisons between results unreliable. Thus, there is a pressing need for standardization and benchmarks that facilitate the equitable comparison of competing approaches. One task where LLMs show great promise is program repair, which can be used to provide debugging support and next-step hints to students. In this article, we propose a novel educational program repair benchmark. We curate two high-quality publicly available programming…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Resource Development and Performance Evaluation
MethodsSparse Evolutionary Training
