ZIGNeRF: Zero-shot 3D Scene Representation with Invertible Generative Neural Radiance Fields

TL;DR

ZIGNeRF enables zero-shot 3D scene reconstruction from a single out-of-domain image by inverting GANs and disentangling objects from backgrounds, allowing 3D manipulations without prior 3D data.

Contribution

The paper introduces ZIGNeRF, a novel model that performs zero-shot GAN inversion for 3D scene representation from a single image, including object-background disentanglement and 3D transformations.

Findings

Successfully generates multi-view images from a single out-of-domain image.

Capable of disentangling objects from backgrounds.

Performs 3D rotations and translations on single images.

Abstract

Generative Neural Radiance Fields (NeRFs) have demonstrated remarkable proficiency in synthesizing multi-view images by learning the distribution of a set of unposed images. Despite the aptitude of existing generative NeRFs in generating 3D-consistent high-quality random samples within data distribution, the creation of a 3D representation of a singular input image remains a formidable challenge. In this manuscript, we introduce ZIGNeRF, an innovative model that executes zero-shot Generative Adversarial Network (GAN) inversion for the generation of multi-view images from a single out-of-domain image. The model is underpinned by a novel inverter that maps out-of-domain images into the latent code of the generator manifold. Notably, ZIGNeRF is capable of disentangling the object from the background and executing 3D operations such as 360-degree rotation or depth and horizontal translation. The efficacy of our model is validated using multiple real-image datasets: Cats, AFHQ, CelebA, CelebA-HQ, and CompCars.