Program Synthesis from Visual Specification
Evan Hernandez, Ara Vartanian, and Xiaojin Zhu

TL;DR
This paper introduces a novel program synthesis approach that generates code from noisy, sketch-like user drawings, enabling automatic completion of simple drawing programs based on visual specifications.
Contribution
It formulates a new synthesis setting using visual sketches as specifications and develops algorithms that can optimally generate programs matching these noisy inputs.
Findings
Algorithms successfully synthesize programs matching user drawings.
Hausdorff distance effectively measures similarity between output and sketch.
Approach works with real user data in a practical setting.
Abstract
Program synthesis is the process of automatically translating a specification into computer code. Traditional synthesis settings require a formal, precise specification. Motivated by computer education applications where a student learns to code simple turtle-style drawing programs, we study a novel synthesis setting where only a noisy user-intention drawing is specified. This allows students to sketch their intended output, optionally together with their own incomplete program, to automatically produce a completed program. We formulate this synthesis problem as search in the space of programs, with the score of a state being the Hausdorff distance between the program output and the user drawing. We compare several search algorithms on a corpus consisting of real user drawings and the corresponding programs, and demonstrate that our algorithms can synthesize programs optimally…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Teaching and Learning Programming · Software Testing and Debugging Techniques
