Nerfies: Deformable Neural Radiance Fields

TL;DR

Nerfies introduces a novel method for photorealistic reconstruction of deformable scenes from casual mobile phone photos/videos, using an augmented neural radiance field with deformation modeling and robust optimization techniques.

Contribution

The paper presents the first deformable NeRF model with a coarse-to-fine optimization and elastic regularization, enabling casual scene reconstruction from mobile videos.

Findings

Successfully reconstructs deforming scenes from casual videos.

Achieves high-fidelity novel view synthesis of non-rigid scenes.

Demonstrates robustness with a new regularization approach.

Abstract

We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity.