TestSelector: Automatic Test Suite Selection for Student Projects -- Extended Version

TL;DR

TestSelector is an extensible framework that automates the selection of optimal test suites for student programming projects, improving bug detection over random selection by supporting complex coverage measures.

Contribution

It introduces a flexible, extensible framework for automatic test suite selection that does not rely on encoding coverage measures into constraint solvers.

Findings

Selected test suites outperform random ones in bug detection

Supports a range of classical code coverage measures

Effective for real-world algorithms projects

Abstract

Computer Science course instructors routinely have to create comprehensive test suites to assess programming assignments. The creation of such test suites is typically not trivial as it involves selecting a limited number of tests from a set of (semi-)randomly generated ones. Manual strategies for test selection do not scale when considering large testing inputs needed, for instance, for the assessment of algorithms exercises. To facilitate this process, we present TestSelector, a new framework for automatic selection of optimal test suites for student projects. The key advantage of TestSelector over existing approaches is that it is easily extensible with arbitrarily complex code coverage measures, not requiring these measures to be encoded into the logic of an exact constraint solver. We demonstrate the flexibility of TestSelector by extending it with support for a range of classical code coverage measures and using it to select test suites for a number of real-world algorithms projects, further showing that the selected test suites outperform randomly selected ones in finding bugs in students' code.