Unsupervised Image Transformation Learning via Generative Adversarial Networks

TL;DR

This paper introduces a GAN-based framework for unsupervised learning of image transformations, enabling continuous semantic editing and understanding of transformation spaces from unlabeled images.

Contribution

The proposed method learns a shared transformation space in GANs, allowing continuous image editing and analysis of transformation properties without labeled data.

Findings

Successfully transfers image styles and contents using the learned transformation space.

Enables semantic image editing such as changing seasons or day/night effects.

Provides insights into the organization of transformation factors within the learned space.

Abstract

In this work, we study the image transformation problem, which targets at learning the underlying transformations (e.g., the transition of seasons) from a collection of unlabeled images. However, there could be countless of transformations in the real world, making such a task incredibly challenging, especially under the unsupervised setting. To tackle this obstacle, we propose a novel learning framework built on generative adversarial networks (GANs), where the discriminator and the generator share a transformation space. After the model gets fully optimized, any two points within the shared space are expected to define a valid transformation. In this way, at the inference stage, we manage to adequately extract the variation factor between a customizable image pair by projecting both images onto the transformation space. The resulting transformation vector can further guide the image synthesis, facilitating image editing with continuous semantic change (e.g., altering summer to winter with fall as the intermediate step). Noticeably, the learned transformation space supports not only transferring image styles (e.g., changing day to night), but also manipulating image contents (e.g., adding clouds in the sky). In addition, we make in-depth analysis on the properties of the transformation space to help understand how various transformations are organized. Project page is at https://genforce.github.io/trgan/.