Amortizing Pragmatic Program Synthesis with Rankings

TL;DR

This paper introduces a global pragmatic ranking approach to amortize the computational cost of RSA-based program synthesis, enabling faster and more effective pragmatic synthesis in multi-demonstration settings.

Contribution

It proposes a novel global ranking method that exactly replicates RSA responses for single demonstrations and approximates RSA in multi-demonstration scenarios, significantly improving efficiency.

Findings

Achieves orders of magnitude speed-up over RSA synthesizer.

Outperforms standard non-pragmatic synthesizer.

Effectively approximates RSA in multi-demonstration settings.

Abstract

In program synthesis, an intelligent system takes in a set of user-generated examples and returns a program that is logically consistent with these examples. The usage of Rational Speech Acts (RSA) framework has been successful in building \emph{pragmatic} program synthesizers that return programs which -- in addition to being logically consistent -- account for the fact that a user chooses their examples informatively. However, the computational burden of running the RSA algorithm has restricted the application of pragmatic program synthesis to domains with a small number of possible programs. This work presents a novel method of amortizing the RSA algorithm by leveraging a \emph{global pragmatic ranking} -- a single, total ordering of all the hypotheses. We prove that for a pragmatic synthesizer that uses a single demonstration, our global ranking method exactly replicates RSA's ranked responses. We further empirically show that global rankings effectively approximate the full pragmatic synthesizer in an online, multi-demonstration setting. Experiments on two program synthesis domains using our pragmatic ranking method resulted in orders of magnitudes of speed ups compared to the RSA synthesizer, while outperforming the standard, non-pragmatic synthesizer.