Deep Manifold Traversal: Changing Labels with Convolutional Features

TL;DR

Deep manifold traversal is a versatile, data-driven method for semantic image editing that morphs images along learned natural image manifolds to change labels like age, season, or time of day.

Contribution

It introduces a general, data-driven approach to label changing in images by approximating the natural image manifold and traversing it to achieve semantic transformations.

Findings

Effective across diverse label changing tasks

Requires only example images from source and target domains

Maintains naturalness of images during transformation

Abstract

Many tasks in computer vision can be cast as a "label changing" problem, where the goal is to make a semantic change to the appearance of an image or some subject in an image in order to alter the class membership. Although successful task-specific methods have been developed for some label changing applications, to date no general purpose method exists. Motivated by this we propose deep manifold traversal, a method that addresses the problem in its most general form: it first approximates the manifold of natural images then morphs a test image along a traversal path away from a source class and towards a target class while staying near the manifold throughout. The resulting algorithm is surprisingly effective and versatile. It is completely data driven, requiring only an example set of images from the desired source and target domains. We demonstrate deep manifold traversal on highly diverse label changing tasks: changing an individual's appearance (age and hair color), changing the season of an outdoor image, and transforming a city skyline towards nighttime.