Automated Feedback Generation for Introductory Programming Assignments

TL;DR

This paper introduces an automated system for providing targeted feedback on introductory programming assignments by deriving minimal corrections using reference solutions and error models, improving feedback accuracy.

Contribution

It presents a novel method combining error models and rule-based translation to automatically generate minimal corrections for student code submissions.

Findings

Corrects 65% of incorrect submissions on average

Uses a formal rule-based approach for error correction

Evaluated on real student data from multiple courses

Abstract

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a quantifiable measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong. We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from 6.00 and 6.00x. Our results show that relatively simple error models can correct on average 65% of all incorrect submissions.