Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis

TL;DR

This paper introduces a surface-aligned neural radiance field method for controllable 3D human synthesis from sparse multi-view videos, improving quality and control over body shape and clothing.

Contribution

It presents a novel surface-aligned neural scene representation using mesh surface points and signed distances, addressing an indistinguishability issue with a barycentric projection technique.

Findings

Achieves higher quality novel-view and novel-pose synthesis.

Supports easy control of body shape and clothing.

Outperforms existing methods on ZJU-MoCap and Human3.6M datasets.

Abstract

We propose a new method for reconstructing controllable implicit 3D human models from sparse multi-view RGB videos. Our method defines the neural scene representation on the mesh surface points and signed distances from the surface of a human body mesh. We identify an indistinguishability issue that arises when a point in 3D space is mapped to its nearest surface point on a mesh for learning surface-aligned neural scene representation. To address this issue, we propose projecting a point onto a mesh surface using a barycentric interpolation with modified vertex normals. Experiments with the ZJU-MoCap and Human3.6M datasets show that our approach achieves a higher quality in a novel-view and novel-pose synthesis than existing methods. We also demonstrate that our method easily supports the control of body shape and clothes. Project page: https://pfnet-research.github.io/surface-aligned-nerf/.