Learning to Edit Visual Programs with Self-Supervision

TL;DR

This paper introduces a self-supervised learning system that improves visual program editing by combining a local edit network with a one-shot program predictor, leading to more accurate visual programs across multiple domains.

Contribution

The paper presents a novel self-supervised approach integrating an edit network with a one-shot program predictor, enhancing visual program inference without requiring program annotations.

Findings

Our method outperforms the one-shot model in accuracy within the same search time.

The joint finetuning scheme improves the quality of inferred visual programs.

The editing-based paradigm shows significant advantages across multiple domains.

Abstract

We design a system that learns how to edit visual programs. Our edit network consumes a complete input program and a visual target. From this input, we task our network with predicting a local edit operation that could be applied to the input program to improve its similarity to the target. In order to apply this scheme for domains that lack program annotations, we develop a self-supervised learning approach that integrates this edit network into a bootstrapped finetuning loop along with a network that predicts entire programs in one-shot. Our joint finetuning scheme, when coupled with an inference procedure that initializes a population from the one-shot model and evolves members of this population with the edit network, helps to infer more accurate visual programs. Over multiple domains, we experimentally compare our method against the alternative of using only the one-shot model, and find that even under equal search-time budgets, our editing-based paradigm provides significant advantages.