Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation

TL;DR

This paper introduces an interactive GAN-based sketch-to-image translation system that enables users to iteratively create and refine images of simple objects through a feedback loop of sketch completion and image synthesis.

Contribution

It presents a novel interactive method with a gating-based class conditioning approach allowing a single model to generate multiple object classes without feature mixing.

Findings

Supports multiple object classes with a single model

Provides real-time interactive sketch completion and image synthesis

Enables novice users to create images through iterative feedback

Abstract

We propose an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects. As the user starts to draw a sketch of a desired object type, the network interactively recommends plausible completions, and shows a corresponding synthesized image to the user. This enables a feedback loop, where the user can edit their sketch based on the network's recommendations, visualizing both the completed shape and final rendered image while they draw. In order to use a single trained model across a wide array of object classes, we introduce a gating-based approach for class conditioning, which allows us to generate distinct classes without feature mixing, from a single generator network. Video available at our website: https://arnabgho.github.io/iSketchNFill/.