Program Synthesis from Visual Specification

TL;DR

This paper introduces a novel program synthesis approach that generates code from noisy, sketch-like user drawings, enabling automatic completion of simple drawing programs based on visual specifications.

Contribution

It formulates a new synthesis setting using visual sketches as specifications and develops algorithms that can optimally generate programs matching these noisy inputs.

Findings

Algorithms successfully synthesize programs matching user drawings.

Hausdorff distance effectively measures similarity between output and sketch.

Approach works with real user data in a practical setting.

Abstract

Program synthesis is the process of automatically translating a specification into computer code. Traditional synthesis settings require a formal, precise specification. Motivated by computer education applications where a student learns to code simple turtle-style drawing programs, we study a novel synthesis setting where only a noisy user-intention drawing is specified. This allows students to sketch their intended output, optionally together with their own incomplete program, to automatically produce a completed program. We formulate this synthesis problem as search in the space of programs, with the score of a state being the Hausdorff distance between the program output and the user drawing. We compare several search algorithms on a corpus consisting of real user drawings and the corresponding programs, and demonstrate that our algorithms can synthesize programs optimally satisfying the specification.