LatentKeypointGAN: Controlling Images via Latent Keypoints -- Extended Abstract

TL;DR

LatentKeypointGAN introduces a self-supervised two-stage GAN that enables detailed control over image content through interpretable latent keypoints, allowing flexible editing and combination of image parts across diverse domains.

Contribution

It presents a novel self-supervised framework that disentangles spatial and appearance factors using latent keypoints, facilitating controllable image generation without labels.

Findings

Provides an interpretable latent space for image editing.

Enables re-arrangement and combination of image parts.

Works across diverse application domains.

Abstract

Generative adversarial networks (GANs) can now generate photo-realistic images. However, how to best control the image content remains an open challenge. We introduce LatentKeypointGAN, a two-stage GAN internally conditioned on a set of keypoints and associated appearance embeddings providing control of the position and style of the generated objects and their respective parts. A major difficulty that we address is disentangling the image into spatial and appearance factors with little domain knowledge and supervision signals. We demonstrate in a user study and quantitative experiments that LatentKeypointGAN provides an interpretable latent space that can be used to re-arrange the generated images by re-positioning and exchanging keypoint embeddings, such as generating portraits by combining the eyes, and mouth from different images. Notably, our method does not require labels as it is self-supervised and thereby applies to diverse application domains, such as editing portraits, indoor rooms, and full-body human poses.