Diffusion On Syntax Trees For Program Synthesis

TL;DR

This paper introduces neural diffusion models operating on syntax trees for program synthesis, enabling iterative editing of code with maintained syntax validity, and demonstrates their effectiveness in inverse graphics and sketch-based program generation.

Contribution

It presents a novel diffusion-based approach on syntax trees for program synthesis, integrating editing and search to improve code generation and debugging.

Findings

Effective in inverse graphics tasks converting images to programs

Able to generate graphics programs from hand-drawn sketches

Enhances program synthesis with syntax-preserving iterative editing

Abstract

Large language models generate code one token at a time. Their autoregressive generation process lacks the feedback of observing the program's output. Training LLMs to suggest edits directly can be challenging due to the scarcity of rich edit data. To address these problems, we propose neural diffusion models that operate on syntax trees of any context-free grammar. Similar to image diffusion models, our method also inverts ``noise'' applied to syntax trees. Rather than generating code sequentially, we iteratively edit it while preserving syntactic validity, which makes it easy to combine this neural model with search. We apply our approach to inverse graphics tasks, where our model learns to convert images into programs that produce those images. Combined with search, our model is able to write graphics programs, see the execution result, and debug them to meet the required specifications. We additionally show how our system can write graphics programs for hand-drawn sketches.