Active Learning for Efficient Testing of Student Programs

TL;DR

This paper introduces ATAS, an automated active learning approach that enhances the efficiency of testing student programs by combining symbolic execution with active learning, significantly reducing runtime while maintaining accuracy.

Contribution

The paper presents ATAS, a novel method that integrates symbolic execution and active learning to improve the speed of semantic bug detection in student programs.

Findings

Over 2.5x faster runtime compared to baseline

Maintains similar evaluation accuracy

Effective on 6 programming tasks with 23K submissions

Abstract

In this work, we propose an automated method to identify semantic bugs in student programs, called ATAS, which builds upon the recent advances in both symbolic execution and active learning. Symbolic execution is a program analysis technique which can generate test cases through symbolic constraint solving. Our method makes use of a reference implementation of the task as its sole input. We compare our method with a symbolic execution-based baseline on 6 programming tasks retrieved from CodeForces comprising a total of 23K student submissions. We show an average improvement of over 2.5x over the baseline in terms of runtime (thus making it more suitable for online evaluation), without a significant degradation in evaluation accuracy.