Benchmarking Educational Program Repair

TL;DR

This paper introduces a standardized benchmark for educational program repair using large language models, including curated datasets, a new evaluation metric, and baseline model performance, to enable fair comparison of approaches.

Contribution

It presents a novel benchmark with curated datasets, a unified evaluation procedure, and a new metric for assessing program repair quality in educational contexts.

Findings

Baseline models show varying repair quality

The rouge@k metric correlates with repair effectiveness

Standardized evaluation enables fair comparison

Abstract

The emergence of large language models (LLMs) has sparked enormous interest due to their potential application across a range of educational tasks. For example, recent work in programming education has used LLMs to generate learning resources, improve error messages, and provide feedback on code. However, one factor that limits progress within the field is that much of the research uses bespoke datasets and different evaluation metrics, making direct comparisons between results unreliable. Thus, there is a pressing need for standardization and benchmarks that facilitate the equitable comparison of competing approaches. One task where LLMs show great promise is program repair, which can be used to provide debugging support and next-step hints to students. In this article, we propose a novel educational program repair benchmark. We curate two high-quality publicly available programming datasets, present a unified evaluation procedure introducing a novel evaluation metric rouge@k for approximating the quality of repairs, and evaluate a set of five recent models to establish baseline performance.