Image Generation and Translation with Disentangled Representations

TL;DR

This paper presents a unified model capable of controllable image generation and multi-domain image translation by disentangling image representations into structured and unstructured parts, enabling flexible manipulation with minimal labeled data.

Contribution

It introduces a novel model that combines controllable generation and translation across multiple domains using disentangled representations, reducing the need for multiple models.

Findings

Able to generate images with specified characteristics

Performs effective image-to-image translation across domains

Learns unknown data factors with minimal supervision

Abstract

Generative models have made significant progress in the tasks of modeling complex data distributions such as natural images. The introduction of Generative Adversarial Networks (GANs) and auto-encoders lead to the possibility of training on big data sets in an unsupervised manner. However, for many generative models it is not possible to specify what kind of image should be generated and it is not possible to translate existing images into new images of similar domains. Furthermore, models that can perform image-to-image translation often need distinct models for each domain, making it hard to scale these systems to multiple domain image-to-image translation. We introduce a model that can do both, controllable image generation and image-to-image translation between multiple domains. We split our image representation into two parts encoding unstructured and structured information respectively. The latter is designed in a disentangled manner, so that different parts encode different image characteristics. We train an encoder to encode images into these representations and use a small amount of labeled data to specify what kind of information should be encoded in the disentangled part. A generator is trained to generate images from these representations using the characteristics provided by the disentangled part of the representation. Through this we can control what kind of images the generator generates, translate images between different domains, and even learn unknown data-generating factors while only using one single model.