Tuning Random Generators: Property-Based Testing as Probabilistic Programming

TL;DR

This paper introduces a method for automatically tuning property-based testing generators using probabilistic programming, leading to improved test input distributions and significantly faster bug detection.

Contribution

It presents a novel approach for automatic, offline tuning of generator weights via probabilistic programming, enhancing distribution control and testing efficiency.

Findings

Optimized generator distributions improve bug detection speed by 3.1-7.4x.

The approach effectively targets desired distributions and enhances test diversity.

Empirical results demonstrate the method's effectiveness in real benchmarks.

Abstract

Property-based testing validates software against an executable specification by evaluating it on randomly generated inputs. The standard way that PBT users generate test inputs is via generators that describe how to sample test inputs through random choices. To achieve a good distribution over test inputs, users must tune their generators, i.e., decide on the weights of these individual random choices. Unfortunately, it is very difficult to understand how to choose individual generator weights in order to achieve a desired distribution, so today this process is tedious and limits the distributions that can be practically achieved. In this paper, we develop techniques for the automatic and offline tuning of generators. Given a generator with undetermined symbolic weights and an objective function, our approach automatically learns values for these weights that optimize for the objective. We describe useful objective functions that allow users to (1) target desired distributions and (2) improve the diversity and validity of their test cases. We have implemented our approach in a novel discrete probabilistic programming system, Loaded Dice, that supports differentiation and parameter learning, and use it as a language for generators. We empirically demonstrate that our approach is effective at optimizing generator distributions according to the specified objective functions. We also perform a thorough evaluation on PBT benchmarks, demonstrating that, when automatically tuned for diversity and validity, the generators exhibit a 3.1-7.4x speedup in bug finding.