Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features

TL;DR

This paper introduces Rayleigh EigenDirections (REDs), a principled method for traversing deep generative model latent spaces to manipulate multiple image features simultaneously while maintaining others, enabling advanced feature control.

Contribution

The paper presents a novel eigenvalue-based approach for multidimensional feature manipulation in GAN latent spaces, extending beyond previous methods.

Findings

Effective control of facial identity and regional pixels.

Capable of manipulating multiple features simultaneously.

Demonstrated on StyleGAN2 for faces and rooms.

Abstract

We present a method for finding paths in a deep generative model's latent space that can maximally vary one set of image features while holding others constant. Crucially, unlike past traversal approaches, ours can manipulate multidimensional features of an image such as facial identity and pixels within a specified region. Our method is principled and conceptually simple: optimal traversal directions are chosen by maximizing differential changes to one feature set such that changes to another set are negligible. We show that this problem is nearly equivalent to one of Rayleigh quotient maximization, and provide a closed-form solution to it based on solving a generalized eigenvalue equation. We use repeated computations of the corresponding optimal directions, which we call Rayleigh EigenDirections (REDs), to generate appropriately curved paths in latent space. We empirically evaluate our method using StyleGAN2 on two image domains: faces and living rooms. We show that our method is capable of controlling various multidimensional features out of the scope of previous latent space traversal methods: face identity, spatial frequency bands, pixels within a region, and the appearance and position of an object. Our work suggests that a wealth of opportunities lies in the local analysis of the geometry and semantics of latent spaces.