Cluster-guided Image Synthesis with Unconditional Models

TL;DR

This paper introduces a method for controllable image synthesis using unsupervised GANs by leveraging cluster structures in the generator's intermediate representations to guide attribute-specific image generation.

Contribution

It reveals that intermediate generator layers form meaningful clusters that can be used to control generated image attributes without supervised labels.

Findings

Clusters correspond to semantic attributes like hair color and pose.

Controlling cluster assignments influences generated image attributes.

Method works across diverse datasets and pre-trained models.

Abstract

Generative Adversarial Networks (GANs) are the driving force behind the state-of-the-art in image generation. Despite their ability to synthesize high-resolution photo-realistic images, generating content with on-demand conditioning of different granularity remains a challenge. This challenge is usually tackled by annotating massive datasets with the attributes of interest, a laborious task that is not always a viable option. Therefore, it is vital to introduce control into the generation process of unsupervised generative models. In this work, we focus on controllable image generation by leveraging GANs that are well-trained in an unsupervised fashion. To this end, we discover that the representation space of intermediate layers of the generator forms a number of clusters that separate the data according to semantically meaningful attributes (e.g., hair color and pose). By conditioning on the cluster assignments, the proposed method is able to control the semantic class of the generated image. Our approach enables sampling from each cluster by Implicit Maximum Likelihood Estimation (IMLE). We showcase the efficacy of our approach on faces (CelebA-HQ and FFHQ), animals (Imagenet) and objects (LSUN) using different pre-trained generative models. The results highlight the ability of our approach to condition image generation on attributes like gender, pose and hair style on faces, as well as a variety of features on different object classes.