Edit Based Grading of SQL Queries

TL;DR

This paper introduces an automated method for grading SQL queries by finding minimal edits to correct student submissions, enabling partial credit and providing targeted feedback, which improves scalability and accuracy in large classes.

Contribution

It proposes a greedy heuristic approach for minimal query edits to facilitate partial marking and feedback, addressing limitations of existing formal equivalence checks.

Findings

Effective partial marking in large classes demonstrated

Greedy heuristic balances runtime and accuracy

System successfully used in IIT courses

Abstract

Grading student SQL queries manually is a tedious and error-prone process. Earlier work on testing correctness of student SQL queries, such as the XData system, can be used to test correctness of a student query. However, in case a student query is found to be incorrect there is currently no way to automatically assign partial marks. Partial marking is important so that small errors are penalized less than large errors. Manually awarding partial marks is not scalable for classes with large number of students, especially MOOCs, and is also prone to human errors. In this paper, we discuss techniques to find a minimum cost set of edits to a student query that would make it correct, which can help assign partial marks, and to help students understand exactly where they went wrong. Given the limitations of current formal methods for checking equivalence, our approach is based on finding nearest query, from a set of instructor provided correct queries, that is found to be equivalent based on query canonicalization. We show that exhaustive techniques are expensive, and propose a greedy heuristic approach that works well both in terms of runtime and accuracy on queries in real-world datasets. Our system can also be used in a learning mode where query edits can be suggested as feedback to students to guide them towards a correct query. Our partial marking system has been successfully used in courses at IIT Bombay and IIT Dharwad.