Program Equivalence for Assisted Grading of Functional Programs (Extended Version)

TL;DR

This paper introduces a formal equivalence algorithm to assist human graders in providing targeted feedback on student programming submissions, significantly reducing grading time while ensuring accuracy.

Contribution

It presents a conservative, formal methods-based equivalence detection technique that scales human grading by identifying similar student programs with guaranteed correctness.

Findings

Reduces grading time for thousands of submissions

Guarantees no incorrect clustering of different algorithms

Effective for functional programming assignments

Abstract

In courses that involve programming assignments, giving meaningful feedback to students is an important challenge. Human beings can give useful feedback by manually grading the programs but this is a time-consuming, labor intensive, and usually boring process. Automatic graders can be fast and scale well but they usually provide poor feedback. Although there has been research on improving automatic graders, research on scaling and improving human grading is limited. We propose to scale human grading by augmenting the manual grading process with an equivalence algorithm that can identify the equivalences between student submissions. This enables human graders to give targeted feedback for multiple student submissions at once. Our technique is conservative in two aspects. First, it identifies equivalence between submissions that are algorithmically similar, e.g., it cannot identify the equivalence between quicksort and mergesort. Second, it uses formal methods instead of clustering algorithms from the machine learning literature. This allows us to prove a soundness result that guarantees that submissions will never be clustered together in error. Despite only reporting equivalence when there is algorithmic similarity and the ability to formally prove equivalence, we show that our technique can significantly reduce grading time for thousands of programming submissions from an introductory functional programming course.