Building Very Small Test Suites (with Snap)

TL;DR

This paper introduces SNAP, a test suite generator that efficiently produces smaller, faster, and fully valid test suites for large, complex software by sampling around average values, outperforming previous methods.

Contribution

The paper presents SNAP, a novel approach combining theorem proving with sampling tactics to generate significantly smaller and faster test suites for large-scale software.

Findings

SNAP produced test suites 10 to 750 times smaller.

SNAP was orders of magnitude faster than previous methods.

SNAP generated 100% valid tests, unlike prior work.

Abstract

Software is now so large and complex that additional architecture is needed to guide theorem provers as they try to generate test suites. For example, the SNAP test suite generator (introduced in this paper) combines the Z3 theorem prover with the following tactic: sample around the average values seen in a few randomly selected valid tests. This tactic is remarkably effective. For 27 real-world programs with up to half a million variables, SNAP found test suites which were 10 to 750 smaller times than those found by the prior state-of-the-art. Also, SNAP ran orders of magnitude faster and (unlike prior work) generated 100% valid tests.