Online Prompt Selection for Program Synthesis
Yixuan Li, Lewis Frampton, Federico Mora, Elizabeth Polgreen

TL;DR
This paper introduces CYANEA, an online learning system that dynamically selects the best solver or prompt combination for program synthesis tasks, significantly improving success rates and efficiency over static approaches.
Contribution
It presents a novel multi-armed bandit based method for adaptive solver and prompt selection in program synthesis, addressing non-expert user challenges.
Findings
CYANEA solves 37.2% more queries than the best single solver.
Achieves results within 4% of the virtual best solver.
Effective across various synthesis benchmarks.
Abstract
Large Language Models (LLMs) demonstrate impressive capabilities in the domain of program synthesis. This level of performance is not, however, universal across all tasks, all LLMs and all prompting styles. There are many areas where one LLM dominates, one prompting style dominates, or where calling a symbolic solver is a better choice than an LLM. A key challenge for the user then, is to identify not only when an LLM is the right choice of solver, and the appropriate LLM to call for a given synthesis task, but also the right way to call it. A non-expert user who makes the wrong choice, incurs a cost both in terms of results (number of tasks solved, and the time it takes to solve them) and financial cost, if using a closed-source language model via a commercial API. We frame this choice as an online learning problem. We use a multi-armed bandit algorithm to select which symbolic solver,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsParallel Computing and Optimization Techniques · Embedded Systems Design Techniques · Logic, programming, and type systems
