NPC: Neural Point Characters from Video

TL;DR

This paper introduces a hybrid point-based neural representation for animatable 3D human and animal models from videos, eliminating the need for expensive templates and improving generalization to new poses.

Contribution

It presents a novel point-based approach that automatically extracts 3D points and learns articulated deformations without explicit surface templates, enhancing flexibility and performance.

Findings

Outperforms prior methods on established benchmarks.

Automatically extracts points for diverse characters.

Achieves comparable results to template-based methods.

Abstract

High-fidelity human 3D models can now be learned directly from videos, typically by combining a template-based surface model with neural representations. However, obtaining a template surface requires expensive multi-view capture systems, laser scans, or strictly controlled conditions. Previous methods avoid using a template but rely on a costly or ill-posed mapping from observation to canonical space. We propose a hybrid point-based representation for reconstructing animatable characters that does not require an explicit surface model, while being generalizable to novel poses. For a given video, our method automatically produces an explicit set of 3D points representing approximate canonical geometry, and learns an articulated deformation model that produces pose-dependent point transformations. The points serve both as a scaffold for high-frequency neural features and an anchor for efficiently mapping between observation and canonical space. We demonstrate on established benchmarks that our representation overcomes limitations of prior work operating in either canonical or in observation space. Moreover, our automatic point extraction approach enables learning models of human and animal characters alike, matching the performance of the methods using rigged surface templates despite being more general. Project website: https://lemonatsu.github.io/npc/