Scaling Neural Program Synthesis with Distribution-based Search

TL;DR

This paper introduces a distribution-based search framework for neural program synthesis, proposing two new algorithms, Heap Search and SQRT Sampling, with proven optimality and scalability benefits.

Contribution

It presents a novel search framework and two algorithms that enhance neural program synthesis, with theoretical guarantees and scalable implementation.

Findings

Heap Search and SQRT Sampling are effective search algorithms.

Both algorithms have proven optimality guarantees.

The methods scale well across parallel computing environments.

Abstract

We consider the problem of automatically constructing computer programs from input-output examples. We investigate how to augment probabilistic and neural program synthesis methods with new search algorithms, proposing a framework called distribution-based search. Within this framework, we introduce two new search algorithms: Heap Search, an enumerative method, and SQRT Sampling, a probabilistic method. We prove certain optimality guarantees for both methods, show how they integrate with probabilistic and neural techniques, and demonstrate how they can operate at scale across parallel compute environments. Collectively these findings offer theoretical and applied studies of search algorithms for program synthesis that integrate with recent developments in machine-learned program synthesizers.