Parsing Randomness: Unifying and Differentiating Parsers and Random Generators

TL;DR

This paper formalizes the relationship between parsers and random data generators using free generators, enabling analysis and new algorithms for generating data structures with constraints.

Contribution

It introduces free generators that unify parsing and generation, and develops derivatives and algorithms for constrained data generation.

Findings

Unified framework for parsers and generators

Proof that many generators can be factored into parsers plus distributions

New algorithm for constrained data structure generation

Abstract

"A generator is a parser of randomness." This perspective on generators for random data structures is well established as folklore in the programming languages community, but it has apparently never been formalized, nor have its consequences been deeply explored. We present free generators, which unify parsing and generation using a common structure that makes the relationship between the two concepts precise. Free generators lead naturally to a proof that a large class of generators can be factored into a parser plus a distribution over choice sequences. Further, free generators support a notion of derivative, analogous to familiar Brzozowski derivatives of formal languages, that allows analysis tools to "preview" the effect of a particular generator choice. This, in turn, gives rise to a novel algorithm for generating data structures satisfying user-specified preconditions.