Image Generation from Sketch Constraint Using Contextual GAN

TL;DR

This paper introduces a novel contextual GAN approach for image generation from sketches, allowing more flexible and realistic outputs by treating sketches as weak constraints in a joint image completion framework.

Contribution

The paper proposes a joint image completion method using a contextual GAN that learns the joint distribution of sketches and images, enabling more flexible image generation from sketches.

Findings

Generates more realistic images than state-of-the-art conditional GANs.

Effectively handles poorly drawn sketches as weak constraints.

Generalizes well across different datasets and categories.

Abstract

In this paper we investigate image generation guided by hand sketch. When the input sketch is badly drawn, the output of common image-to-image translation follows the input edges due to the hard condition imposed by the translation process. Instead, we propose to use sketch as weak constraint, where the output edges do not necessarily follow the input edges. We address this problem using a novel joint image completion approach, where the sketch provides the image context for completing, or generating the output image. We train a generated adversarial network, i.e, contextual GAN to learn the joint distribution of sketch and the corresponding image by using joint images. Our contextual GAN has several advantages. First, the simple joint image representation allows for simple and effective learning of joint distribution in the same image-sketch space, which avoids complicated issues in cross-domain learning. Second, while the output is related to its input overall, the generated features exhibit more freedom in appearance and do not strictly align with the input features as previous conditional GANs do. Third, from the joint image's point of view, image and sketch are of no difference, thus exactly the same deep joint image completion network can be used for image-to-sketch generation. Experiments evaluated on three different datasets show that our contextual GAN can generate more realistic images than state-of-the-art conditional GANs on challenging inputs and generalize well on common categories.