In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing

TL;DR

This paper introduces a 3D GAN inversion method that explicitly models out-of-distribution objects using volumetric decomposition, improving reconstruction and editing of complex face images.

Contribution

It proposes a novel volumetric decomposition approach with separate neural radiance fields for in-distribution and OOD objects, enhancing reconstruction and editability.

Findings

Improved reconstruction accuracy on challenging real face images.

Enhanced editability while maintaining high fidelity.

Favorable comparison against baseline methods.

Abstract

3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. However, a model pre-trained on a particular dataset (e.g., FFHQ) often has difficulty reconstructing images with out-of-distribution (OOD) objects such as faces with heavy make-up or occluding objects. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs. Our core idea is to represent the image using two individual neural radiance fields: one for the in-distribution content and the other for the out-of-distribution object. The final reconstruction is achieved by optimizing the composition of these two radiance fields with carefully designed regularization. We demonstrate that our explicit decomposition alleviates the inherent trade-off between reconstruction fidelity and editability. We evaluate reconstruction accuracy and editability of our method on challenging real face images and videos and showcase favorable results against other baselines.