Improving Unsupervised Visual Program Inference with Code Rewriting Families

TL;DR

This paper introduces SIRI, a framework using code rewriting to enhance unsupervised visual program inference, leading to better reconstructions, faster learning, and more parsimonious programs in 2D and 3D domains.

Contribution

The paper presents a novel code rewriting framework, SIRI, that improves unsupervised visual program inference by enhancing program quality and learning efficiency.

Findings

SIRI improves reconstruction accuracy in 2D and 3D shape programming.

SIRI accelerates convergence rates during training.

Rewriters produce more parsimonious programs with fewer primitives.

Abstract

Programs offer compactness and structure that makes them an attractive representation for visual data. We explore how code rewriting can be used to improve systems for inferring programs from visual data. We first propose Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised bootstrapped learning. SIRI sparsely applies code rewrite operations over a dataset of training programs, injecting the improved programs back into the training set. We design a family of rewriters for visual programming domains: parameter optimization, code pruning, and code grafting. For three shape programming languages in 2D and 3D, we show that using SIRI with our family of rewriters improves performance: better reconstructions and faster convergence rates, compared with bootstrapped learning methods that do not use rewriters or use them naively. Finally, we demonstrate that our family of rewriters can be effectively used at test time to improve the output of SIRI predictions. For 2D and 3D CSG, we outperform or match the reconstruction performance of recent domain-specific neural architectures, while producing more parsimonious programs that use significantly fewer primitives.