Generative Visual Manipulation on the Natural Image Manifold

TL;DR

This paper introduces a method for realistic image manipulation by learning a natural image manifold with GANs and constraining edits to stay on this manifold, enabling user-controlled, realistic modifications in near-real time.

Contribution

It presents a novel approach that combines generative adversarial networks with constrained optimization for realistic, user-controlled image editing on the learned manifold.

Findings

Enables realistic shape and color editing of images.

Allows changing one image to resemble another.

Supports generating new images from user scribbles.

Abstract

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to "fall off" the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user's scribbles.