HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesis

TL;DR

This paper introduces HYSYNTH, a hybrid method combining large language models and symbolic search to improve program synthesis accuracy across multiple domains.

Contribution

It presents a novel context-free surrogate model learned from LLM completions to guide program synthesis, outperforming existing methods.

Findings

Outperforms unguided search and LLM sampling

Effective across three different domains

Improves synthesis accuracy significantly

Abstract

Many structured prediction and reasoning tasks can be framed as program synthesis problems, where the goal is to generate a program in a domain-specific language (DSL) that transforms input data into the desired output. Unfortunately, purely neural approaches, such as large language models (LLMs), often fail to produce fully correct programs in unfamiliar DSLs, while purely symbolic methods based on combinatorial search scale poorly to complex problems. Motivated by these limitations, we introduce a hybrid approach, where LLM completions for a given task are used to learn a task-specific, context-free surrogate model, which is then used to guide program synthesis. We evaluate this hybrid approach on three domains, and show that it outperforms both unguided search and direct sampling from LLMs, as well as existing program synthesizers.