PersonNeRF: Personalized Reconstruction from Photo Collections

TL;DR

PersonNeRF creates personalized 3D models from photo collections of individuals, enabling realistic rendering from new viewpoints, poses, and appearances despite sparse and unstructured data.

Contribution

It introduces a novel canonical T-pose neural volumetric representation that handles appearance changes and pose variations in unstructured photo collections.

Findings

Outperforms prior methods in free-viewpoint human rendering.

Successfully models appearance and pose variations from sparse, unstructured photos.

Produces compelling renderings from novel viewpoint and pose combinations.

Abstract

We present PersonNeRF, a method that takes a collection of photos of a subject (e.g. Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance. PersonNeRF builds a customized neural volumetric 3D model of the subject that is able to render an entire space spanned by camera viewpoint, body pose, and appearance. A central challenge in this task is dealing with sparse observations; a given body pose is likely only observed by a single viewpoint with a single appearance, and a given appearance is only observed under a handful of different body poses. We address this issue by recovering a canonical T-pose neural volumetric representation of the subject that allows for changing appearance across different observations, but uses a shared pose-dependent motion field across all observations. We demonstrate that this approach, along with regularization of the recovered volumetric geometry to encourage smoothness, is able to recover a model that renders compelling images from novel combinations of viewpoint, pose, and appearance from these challenging unstructured photo collections, outperforming prior work for free-viewpoint human rendering.