Spatial Latent Representations in Generative Adversarial Networks for Image Generation

TL;DR

This paper introduces spatial latent spaces for StyleGAN2 that better capture spatial details and semantic information, enabling improved image manipulation, out-of-sample content representation, and high-quality image generation of arbitrary sizes.

Contribution

It proposes a novel family of spatial latent spaces, encoding methods, and training procedures that enhance image editing, quality, and scalability in GANs.

Findings

Encoding quality improved by up to 30% in LPIPS score.

Spatial latent spaces enable out-of-sample object arrangement representation.

Proposed training method improves FID score by 29% on SpaceNet.

Abstract

In the majority of GAN architectures, the latent space is defined as a set of vectors of given dimensionality. Such representations are not easily interpretable and do not capture spatial information of image content directly. In this work, we define a family of spatial latent spaces for StyleGAN2, capable of capturing more details and representing images that are out-of-sample in terms of the number and arrangement of object parts, such as an image of multiple faces or a face with more than two eyes. We propose a method for encoding images into our spaces, together with an attribute model capable of performing attribute editing in these spaces. We show that our spaces are effective for image manipulation and encode semantic information well. Our approach can be used on pre-trained generator models, and attribute edition can be done using pre-generated direction vectors making the barrier to entry for experimentation and use extremely low. We propose a regularization method for optimizing latent representations, which equalizes distributions of parts of latent spaces, making representations much closer to generated ones. We use it for encoding images into spatial spaces to obtain significant improvement in quality while keeping semantics and ability to use our attribute model for edition purposes. In total, using our methods gives encoding quality boost even as high as 30% in terms of LPIPS score comparing to standard methods, while keeping semantics. Additionally, we propose a StyleGAN2 training procedure on our spatial latent spaces, together with a custom spatial latent representation distribution to make spatially closer elements in the representation more dependent on each other than farther elements. Such approach improves the FID score by 29% on SpaceNet, and is able to generate consistent images of arbitrary sizes on spatially homogeneous datasets, like satellite imagery.