Spatio-Temporal Garment Reconstruction Using Diffusion Mapping via Pattern Coordinates

TL;DR

This paper introduces a novel spatio-temporal diffusion framework for high-fidelity 3D garment reconstruction from monocular images and videos, effectively capturing detailed and dynamic clothing geometry.

Contribution

It combines implicit sewing patterns with diffusion models and a mapping approach to improve garment reconstruction accuracy and temporal consistency, especially for loose-fitting clothing.

Findings

Outperforms existing methods on various garment types.

Generalizes well from synthetic training to real-world data.

Preserves fine geometric details and realistic motion.

Abstract

Reconstructing 3D clothed humans from monocular images and videos is a fundamental problem with applications in virtual try-on, avatar creation, and mixed reality. Despite significant progress in human body recovery, accurately reconstructing garment geometry, particularly for loose-fitting clothing, remains an open challenge. We propose a unified framework for high-fidelity 3D garment reconstruction from both single images and video sequences. Our approach combines Implicit Sewing Patterns (ISP) with a generative diffusion model to learn expressive garment shape priors in 2D UV space. Leveraging these priors, we introduce a mapping model that establishes correspondences between image pixels, UV pattern coordinates, and 3D geometry, enabling accurate and detailed garment reconstruction from single images. We further extend this formulation to dynamic reconstruction by introducing a spatio-temporal diffusion scheme with test-time guidance to enforce long-range temporal consistency. We also develop analytic projection-based constraints that preserve image-aligned geometry in visible regions while enforcing coherent completion in occluded areas over time. Although trained exclusively on synthetically simulated cloth data, our method generalizes well to real-world imagery and consistently outperforms existing approaches on both tight- and loose-fitting garments. The reconstructed garments preserve fine geometric detail while exhibiting realistic dynamic motion, supporting downstream applications such as texture editing, garment retargeting, and animation.