Anomaly Detection in Scratch Assignments

TL;DR

This paper improves an anomaly detection method for identifying task-specific bugs in Scratch student assignments by lowering abstraction levels, making bug detection more focused and effective.

Contribution

It introduces a refined anomaly detection approach that operates at a lower abstraction level to better target relevant program parts in Scratch assignments.

Findings

Lower abstraction level improves bug detection accuracy

Anomaly detection reliably finds task-specific bugs

Focused detection reduces false positives

Abstract

For teachers, automated tool support for debugging and assessing their students' programming assignments is a great help in their everyday business. For block-based programming languages which are commonly used to introduce younger learners to programming, testing frameworks and other software analysis tools exist, but require manual work such as writing test suites or formal specifications. However, most of the teachers using languages like Scratch are not trained for or experienced in this kind of task. Linters do not require manual work but are limited to generic bugs and therefore miss potential task-specific bugs in student solutions. In prior work, we proposed the use of anomaly detection to find project-specific bugs in sets of student programming assignments automatically, without any additional manual labour required from the teachers' side. Evaluation on student solutions for typical programming assignments showed that anomaly detection is a reliable way to locate bugs in a data set of student programs. In this paper, we enhance our initial approach by lowering the abstraction level. The results suggest that the lower abstraction level can focus anomaly detection on the relevant parts of the programs.