Amortizing Pragmatic Program Synthesis with Rankings

TL;DR

This paper introduces a method to efficiently approximate pragmatic program synthesis using rankings, significantly speeding up the process while maintaining or improving accuracy in human communication scenarios.

Contribution

It presents a novel amortization technique that distills RSA-based rankings into a global ranking, enabling fast and accurate pragmatic program synthesis.

Findings

Achieved orders of magnitude speedups over exact RSA synthesizer.

More accurate than non-pragmatic synthesizers in human communication tasks.

Exact in the single-example synthesis case.

Abstract

The usage of Rational Speech Acts (RSA) framework has been successful in building \emph{pragmatic} program synthesizers that return programs which, in addition to being logically consistent with user-generated examples, account for the fact that a user chooses their examples informatively. We present a general method of amortizing the slow, exact RSA synthesizer. Our method first query the exact RSA synthesizer to compile a communication dataset. The dataset contains a number of example-dependent rankings of subsets of programs. It then distills a \textit{single} global ranking of all programs as an approximation to every ranking in the dataset. This global ranking is then used at inference time to rank multiple logically consistent candidate programs generated from a fast, non-pragmatic synthesizer. Experiments on two program synthesis domains using our ranking method resulted in orders of magnitudes of speed ups compared to the exact RSA synthesizer, while being more accurate than a non-pragmatic synthesizer when communicating with humans. Finally, we prove that in the special case of synthesis from a single example, this approximation is exact.