DressRecon: Freeform 4D Human Reconstruction from Monocular Video

TL;DR

DressRecon introduces a novel method for reconstructing detailed, time-consistent 3D human models from monocular videos, effectively handling loose clothing and object interactions without requiring multi-view setups or personalized templates.

Contribution

It combines generic human priors with video-specific deformation models using neural implicit representations for flexible, high-quality 3D human reconstruction from monocular videos.

Findings

Outperforms prior methods on challenging clothing and object interaction datasets.

Produces high-fidelity, time-consistent 3D meshes suitable for rendering.

Leverages image-based priors for capturing subtle clothing geometry.

Abstract

We present a method to reconstruct time-consistent human body models from monocular videos, focusing on extremely loose clothing or handheld object interactions. Prior work in human reconstruction is either limited to tight clothing with no object interactions, or requires calibrated multi-view captures or personalized template scans which are costly to collect at scale. Our key insight for high-quality yet flexible reconstruction is the careful combination of generic human priors about articulated body shape (learned from large-scale training data) with video-specific articulated "bag-of-bones" deformation (fit to a single video via test-time optimization). We accomplish this by learning a neural implicit model that disentangles body versus clothing deformations as separate motion model layers. To capture subtle geometry of clothing, we leverage image-based priors such as human body pose, surface normals, and optical flow during optimization. The resulting neural fields can be extracted into time-consistent meshes, or further optimized as explicit 3D Gaussians for high-fidelity interactive rendering. On datasets with highly challenging clothing deformations and object interactions, DressRecon yields higher-fidelity 3D reconstructions than prior art. Project page: https://jefftan969.github.io/dressrecon/