Bridging the Gap between Unit Test Generation and System Test Generation

TL;DR

This paper introduces a test generation bridge that combines the speed of unit testing with the accuracy of system testing by carving parameterized unit tests from system executions, enabling more effective bug detection.

Contribution

It proposes a novel framework that links system and unit test generation, allowing symbolic exploration of functions within recorded system contexts.

Findings

Higher coverage than state-of-the-art system test generators

Effective identification of true alarms through parameter lifting

Successful application to complex systems like Web/Python/SQLite/C stack

Abstract

Common test generators fall into two categories. Generating test inputs at the unit level is fast, but can lead to false alarms when a function is called with inputs that would not occur in a system context. If a generated input at the system level causes a failure, this is a true alarm, as the input could also have come from the user or a third party; but system testing is much slower. In this paper, we introduce the concept of a test generation bridge, which joins the accuracy of system testing with the speed of unit testing. A Test Generation Bridge allows to combine an arbitrary system test generator with an arbitrary unit test generator. It does so by carving parameterized unit tests from system (test) executions. These unit tests run in a context recorded from the system test, but individual parameters are left free for the unit test generator to systematically explore. This allows symbolic test generators such as KLEE to operate on individual functions in the recorded system context. If the test generator detects a failure, we lift the failure-inducing parameter back to the system input; if the failure can be reproduced at the system level, it is reported as a true alarm. Our BASILISK prototype can extract and test units out of complex systems such as a Web/Python/SQLite/C stack; in its evaluation, it achieves a higher coverage than a state-of-the-art system test generator.