Video Based Reconstruction of 3D People Models

TL;DR

This paper introduces a robust method for reconstructing accurate 3D human models and textures from a single monocular video, enabling accessible creation of animatable digital doubles for various applications.

Contribution

The main novelty is a nonrigid silhouette deformation technique that produces a visual hull for precise 3D surface reconstruction from monocular videos.

Findings

Achieves 5mm accuracy in 3D model fitting.

Works with clothed people and dynamic silhouettes.

Enables creation of fully animatable digital doubles.

Abstract

This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving. Based on a parametric body model, we present a robust processing pipeline achieving 3D model fits with 5mm accuracy also for clothed people. Our main contribution is a method to nonrigidly deform the silhouette cones corresponding to the dynamic human silhouettes, resulting in a visual hull in a common reference frame that enables surface reconstruction. This enables efficient estimation of a consensus 3D shape, texture and implanted animation skeleton based on a large number of frames. We present evaluation results for a number of test subjects and analyze overall performance. Requiring only a smartphone or webcam, our method enables everyone to create their own fully animatable digital double, e.g., for social VR applications or virtual try-on for online fashion shopping.