Input-Gen: Guided Generation of Stateful Inputs for Testing, Tuning, and Training

TL;DR

Input-Gen automatically creates stateful, executable inputs for software testing, tuning, and training, significantly enhancing code coverage and enabling better machine learning models that understand real-world program behavior.

Contribution

We introduce Input-Gen, a compiler-based tool that automatically generates stateful inputs for arbitrary programs, improving testing and training data quality.

Findings

Generated inputs achieved 90% validity on the dataset

Single input yielded 37% code coverage on average

Guided generation of five inputs increased coverage to 45%

Abstract

The size and complexity of software applications is increasing at an accelerating pace. Source code repositories (along with their dependencies) require vast amounts of labor to keep them tested, maintained, and up to date. As the discipline now begins to also incorporate automatically generated programs, automation in testing and tuning is required to keep up with the pace - let alone reduce the present level of complexity. While machine learning has been used to understand and generate code in various contexts, machine learning models themselves are trained almost exclusively on static code without inputs, traces, or other execution time information. This lack of training data limits the ability of these models to understand real-world problems in software. In this work we show that inputs, like code, can be generated automatically at scale. Our generated inputs are stateful, and appear to faithfully reproduce the arbitrary data structures and system calls required to rerun a program function. By building our tool within the compiler, it both can be applied to arbitrary programming languages and architectures and can leverage static analysis and transformations for improved performance. Our approach is able to produce valid inputs, including initial memory states, for 90% of the ComPile dataset modules we explored, for a total of 21.4 million executable functions. Further, we find that a single generated input results in an average block coverage of 37%, whereas guided generation of five inputs improves it to 45%.