Controllable GAN Synthesis Using Non-Rigid Structure-from-Motion

TL;DR

This paper introduces a method that combines non-rigid structure-from-motion with deep generative models to enable 3D shape and camera view editing directly from 2D GAN latent spaces without retraining.

Contribution

It presents a novel framework for discovering latent trajectories corresponding to 3D geometry changes, allowing explicit control over non-rigid shapes and camera parameters in GAN-generated images.

Findings

Enables editing of 3D shape and camera view from 2D GANs.

Provides implicit dense 3D reconstruction from latent space.

Demonstrates effectiveness on face datasets.

Abstract

In this paper, we present an approach for combining non-rigid structure-from-motion (NRSfM) with deep generative models,and propose an efficient framework for discovering trajectories in the latent space of 2D GANs corresponding to changes in 3D geometry. Our approach uses recent advances in NRSfM and enables editing of the camera and non-rigid shape information associated with the latent codes without needing to retrain the generator. This formulation provides an implicit dense 3D reconstruction as it enables the image synthesis of novel shapes from arbitrary view angles and non-rigid structure. The method is built upon a sparse backbone, where a neural regressor is first trained to regress parameters describing the cameras and sparse non-rigid structure directly from the latent codes. The latent trajectories associated with changes in the camera and structure parameters are then identified by estimating the local inverse of the regressor in the neighborhood of a given latent code. The experiments show that our approach provides a versatile, systematic way to model, analyze, and edit the geometry and non-rigid structures of faces.