Learning Implicit Body Representations from Double Diffusion Based Neural Radiance Fields

TL;DR

This paper introduces DD-NeRF, a double diffusion neural radiance field that reconstructs human body geometry and appearance from sparse images, leveraging multi-level priors and diffusion to improve novel view synthesis.

Contribution

The paper proposes a novel double diffusion mechanism for neural radiance fields that effectively models coarse and fine human body details from limited input views.

Findings

Outperforms state-of-the-art in geometric reconstruction

Achieves superior novel view synthesis quality

Effectively models clothing and hair details

Abstract

In this paper, we present a novel double diffusion based neural radiance field, dubbed DD-NeRF, to reconstruct human body geometry and render the human body appearance in novel views from a sparse set of images. We first propose a double diffusion mechanism to achieve expressive representations of input images by fully exploiting human body priors and image appearance details at two levels. At the coarse level, we first model the coarse human body poses and shapes via an unclothed 3D deformable vertex model as guidance. At the fine level, we present a multi-view sampling network to capture subtle geometric deformations and image detailed appearances, such as clothing and hair, from multiple input views. Considering the sparsity of the two level features, we diffuse them into feature volumes in the canonical space to construct neural radiance fields. Then, we present a signed distance function (SDF) regression network to construct body surfaces from the diffused features. Thanks to our double diffused representations, our method can even synthesize novel views of unseen subjects. Experiments on various datasets demonstrate that our approach outperforms the state-of-the-art in both geometric reconstruction and novel view synthesis.