AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture

TL;DR

AvatarCap introduces a novel framework that uses animatable avatars and monocular video to achieve high-fidelity 3D human reconstructions with detailed, pose-dependent dynamics, even in invisible regions.

Contribution

The paper presents AvatarCap, a new approach combining avatar priors and monocular video for detailed 3D human capture, including invisible regions, with a novel GeoTexAvatar model.

Findings

Outperforms state-of-the-art methods in monocular volumetric capture.

Successfully reconstructs detailed 3D models with pose-dependent dynamics.

Enables high-fidelity capture from limited 3D scans and monocular videos.

Abstract

To address the ill-posed problem caused by partial observations in monocular human volumetric capture, we present AvatarCap, a novel framework that introduces animatable avatars into the capture pipeline for high-fidelity reconstruction in both visible and invisible regions. Our method firstly creates an animatable avatar for the subject from a small number (~20) of 3D scans as a prior. Then given a monocular RGB video of this subject, our method integrates information from both the image observation and the avatar prior, and accordingly recon-structs high-fidelity 3D textured models with dynamic details regardless of the visibility. To learn an effective avatar for volumetric capture from only few samples, we propose GeoTexAvatar, which leverages both geometry and texture supervisions to constrain the pose-dependent dynamics in a decomposed implicit manner. An avatar-conditioned volumetric capture method that involves a canonical normal fusion and a reconstruction network is further proposed to integrate both image observations and avatar dynamics for high-fidelity reconstruction in both observed and invisible regions. Overall, our method enables monocular human volumetric capture with detailed and pose-dependent dynamics, and the experiments show that our method outperforms state of the art. Code is available at https://github.com/lizhe00/AvatarCap.