3D GAN Inversion with Pose Optimization

TL;DR

This paper presents a novel 3D GAN inversion method that simultaneously infers camera pose and latent code, enabling multi-view consistent editing and 3D reconstruction from a single image, leveraging pre-trained estimators and NeRF-derived depth.

Contribution

It introduces a generalizable approach for 3D GAN inversion that does not rely on ground-truth camera viewpoints, improving multi-view consistency and image reconstruction quality.

Findings

Enhanced multi-view consistent image editing.

Improved 3D reconstruction accuracy.

Superior results compared to 2D GAN-based editing.

Abstract

With the recent advances in NeRF-based 3D aware GANs quality, projecting an image into the latent space of these 3D-aware GANs has a natural advantage over 2D GAN inversion: not only does it allow multi-view consistent editing of the projected image, but it also enables 3D reconstruction and novel view synthesis when given only a single image. However, the explicit viewpoint control acts as a main hindrance in the 3D GAN inversion process, as both camera pose and latent code have to be optimized simultaneously to reconstruct the given image. Most works that explore the latent space of the 3D-aware GANs rely on ground-truth camera viewpoint or deformable 3D model, thus limiting their applicability. In this work, we introduce a generalizable 3D GAN inversion method that infers camera viewpoint and latent code simultaneously to enable multi-view consistent semantic image editing. The key to our approach is to leverage pre-trained estimators for better initialization and utilize the pixel-wise depth calculated from NeRF parameters to better reconstruct the given image. We conduct extensive experiments on image reconstruction and editing both quantitatively and qualitatively, and further compare our results with 2D GAN-based editing to demonstrate the advantages of utilizing the latent space of 3D GANs. Additional results and visualizations are available at https://3dgan-inversion.github.io .