Learning to Infer Graphics Programs from Hand-Drawn Images

TL;DR

This paper presents a model that combines deep learning and program synthesis to convert hand-drawn images into interpretable graphics programs, enabling error correction, similarity measurement, and extrapolation.

Contribution

It introduces a novel approach that integrates neural networks with program synthesis to infer structured graphics programs from sketches.

Findings

Successfully converts hand drawings into graphics programs

Enables correction and refinement of generated programs

Facilitates measuring similarity and extrapolating drawings

Abstract

We introduce a model that learns to convert simple hand drawings into graphics programs written in a subset of \LaTeX. The model combines techniques from deep learning and program synthesis. We learn a convolutional neural network that proposes plausible drawing primitives that explain an image. These drawing primitives are like a trace of the set of primitive commands issued by a graphics program. We learn a model that uses program synthesis techniques to recover a graphics program from that trace. These programs have constructs like variable bindings, iterative loops, or simple kinds of conditionals. With a graphics program in hand, we can correct errors made by the deep network, measure similarity between drawings by use of similar high-level geometric structures, and extrapolate drawings. Taken together these results are a step towards agents that induce useful, human-readable programs from perceptual input.