NeRFInvertor: High Fidelity NeRF-GAN Inversion for Single-shot Real Image Animation

TL;DR

NeRFInvertor introduces a universal fine-tuning approach for NeRF-GANs that enables high-fidelity, 3D consistent animation of real faces from a single image, overcoming inversion challenges.

Contribution

The paper presents a novel fine-tuning method for NeRF-GANs that improves real face image inversion and animation from a single image, using 2D and 3D regularizations.

Findings

Effective high-fidelity face animation from a single image

Reduces identity gap with 2D loss functions

Removes artifacts via 3D regularizations

Abstract

Nerf-based Generative models have shown impressive capacity in generating high-quality images with consistent 3D geometry. Despite successful synthesis of fake identity images randomly sampled from latent space, adopting these models for generating face images of real subjects is still a challenging task due to its so-called inversion issue. In this paper, we propose a universal method to surgically fine-tune these NeRF-GAN models in order to achieve high-fidelity animation of real subjects only by a single image. Given the optimized latent code for an out-of-domain real image, we employ 2D loss functions on the rendered image to reduce the identity gap. Furthermore, our method leverages explicit and implicit 3D regularizations using the in-domain neighborhood samples around the optimized latent code to remove geometrical and visual artifacts. Our experiments confirm the effectiveness of our method in realistic, high-fidelity, and 3D consistent animation of real faces on multiple NeRF-GAN models across different datasets.