D-NeRF: Neural Radiance Fields for Dynamic Scenes

TL;DR

D-NeRF extends neural radiance fields to dynamic scenes by incorporating time as an input, enabling the reconstruction and rendering of moving objects from a single camera view, with applications to rigid and non-rigid motions.

Contribution

This paper introduces a novel method that models dynamic scenes with neural radiance fields by learning a canonical scene representation and its deformation over time.

Findings

Successfully reconstructs dynamic scenes with rigid and non-rigid motions.

Enables novel view synthesis and object movement control from a single camera.

Demonstrates effectiveness on various complex dynamic scenes.

Abstract

Neural rendering techniques combining machine learning with geometric reasoning have arisen as one of the most promising approaches for synthesizing novel views of a scene from a sparse set of images. Among these, stands out the Neural radiance fields (NeRF), which trains a deep network to map 5D input coordinates (representing spatial location and viewing direction) into a volume density and view-dependent emitted radiance. However, despite achieving an unprecedented level of photorealism on the generated images, NeRF is only applicable to static scenes, where the same spatial location can be queried from different images. In this paper we introduce D-NeRF, a method that extends neural radiance fields to a dynamic domain, allowing to reconstruct and render novel images of objects under rigid and non-rigid motions from a \emph{single} camera moving around the scene. For this purpose we consider time as an additional input to the system, and split the learning process in two main stages: one that encodes the scene into a canonical space and another that maps this canonical representation into the deformed scene at a particular time. Both mappings are simultaneously learned using fully-connected networks. Once the networks are trained, D-NeRF can render novel images, controlling both the camera view and the time variable, and thus, the object movement. We demonstrate the effectiveness of our approach on scenes with objects under rigid, articulated and non-rigid motions. Code, model weights and the dynamic scenes dataset will be released.