Semantically Decomposing the Latent Spaces of Generative Adversarial Networks

TL;DR

This paper introduces a novel training algorithm for GANs that learns disentangled latent spaces for identity and observation, enabling controlled image generation and manipulation.

Contribution

It presents a pairwise training scheme with Siamese discriminators that improves the generation of identity-consistent, photorealistic image pairs in GANs.

Findings

Generated images are convincingly identity-matched according to human judges.

The method enables independent control over identity and observation factors.

Experiments demonstrate improved realism and consistency in generated image pairs.

Abstract

We propose a new algorithm for training generative adversarial networks that jointly learns latent codes for both identities (e.g. individual humans) and observations (e.g. specific photographs). By fixing the identity portion of the latent codes, we can generate diverse images of the same subject, and by fixing the observation portion, we can traverse the manifold of subjects while maintaining contingent aspects such as lighting and pose. Our algorithm features a pairwise training scheme in which each sample from the generator consists of two images with a common identity code. Corresponding samples from the real dataset consist of two distinct photographs of the same subject. In order to fool the discriminator, the generator must produce pairs that are photorealistic, distinct, and appear to depict the same individual. We augment both the DCGAN and BEGAN approaches with Siamese discriminators to facilitate pairwise training. Experiments with human judges and an off-the-shelf face verification system demonstrate our algorithm's ability to generate convincing, identity-matched photographs.