High-Performance Generation of Constrained Inputs

TL;DR

This paper introduces a highly efficient evolutionary approach for constrained input generation in language-based testing, achieving 3-4 orders of magnitude speedup over previous methods and enabling complex constraint solving.

Contribution

The paper presents a novel method combining Rust-based grammar transformations and advanced evolutionary algorithms to significantly improve constrained input generation performance.

Findings

Achieves 3-4 orders of magnitude speedup over prior methods

Successfully generates complex, valid test inputs for a C compiler

Handles constraints that previous methods could not solve

Abstract

Language-based testing combines context-free grammar definitions with semantic constraints over grammar elements to generate test inputs. By pairing context-free grammars with constraints, users have the expressiveness of unrestricted grammars while retaining simple structure. However, producing inputs in the presence of such constraints can be challenging. In past approaches, SMT solvers have been found to be very slow at finding string solutions; evolutionary algorithms are faster and more general, but current implementations still struggle with complex constraints that would be required for domains such as compiler testing. In this paper, we present a novel approach for evolutionary language-based testing that improves performance by 3-4 orders of magnitude over the current state of the art, reducing hours of generation and constraint solving time to seconds. We accomplish this by (1) carefully transforming grammar definitions into Rust types and trait implementations, ensuring that the compiler may near-maximally optimize arbitrary operations on arbitrary grammars; and (2) using better evolutionary algorithms that improve the ability of language-based testing to solve complex constraint systems. These performance and algorithmic improvements allow our prototype, FANDANGO-RS, to solve constraints that previous strategies simply cannot handle. We demonstrate this by a case study for a C subset, in which FANDANGO-RS is able to generate 401 diverse, complex, and valid test inputs for a C compiler per minute.