H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion

TL;DR

H-NeRF introduces a neural radiance field model for rendering and 4D reconstruction of moving humans using sparse or monocular camera data, leveraging implicit human models and geometric constraints for robustness and generalization.

Contribution

The paper proposes a novel H-NeRF framework that integrates structured implicit human models with neural radiance fields, enabling robust, accurate, and generalizable human motion reconstruction from limited views.

Findings

Robust reconstruction from sparse views and monocular videos.

High accuracy in rendering and temporal reconstruction of humans.

Good generalization beyond training poses and views.

Abstract

We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.