USR: Unsupervised Separated 3D Garment and Human Reconstruction via Geometry and Semantic Consistency

TL;DR

This paper introduces USR, an unsupervised method for separately reconstructing 3D human bodies and garments from images, improving realism and enabling applications like virtual try-on without requiring 3D models.

Contribution

The paper presents a novel unsupervised layered reconstruction approach using a surface-aware neural radiance field and semantic-guided segmentation, eliminating the need for 3D supervision.

Findings

Outperforms state-of-the-art in geometry and appearance accuracy

Supports real-time reconstruction of unseen people

Enables clothing swapping and virtual try-on

Abstract

Dressed people reconstruction from images is a popular task with promising applications in the creative media and game industry. However, most existing methods reconstruct the human body and garments as a whole with the supervision of 3D models, which hinders the downstream interaction tasks and requires hard-to-obtain data. To address these issues, we propose an unsupervised separated 3D garments and human reconstruction model (USR), which reconstructs the human body and authentic textured clothes in layers without 3D models. More specifically, our method proposes a generalized surface-aware neural radiance field to learn the mapping between sparse multi-view images and geometries of the dressed people. Based on the full geometry, we introduce a Semantic and Confidence Guided Separation strategy (SCGS) to detect, segment, and reconstruct the clothes layer, leveraging the consistency between 2D semantic and 3D geometry. Moreover, we propose a Geometry Fine-tune Module to smooth edges. Extensive experiments on our dataset show that comparing with state-of-the-art methods, USR achieves improvements on both geometry and appearance reconstruction while supporting generalizing to unseen people in real time. Besides, we also introduce SMPL-D model to show the benefit of the separated modeling of clothes and the human body that allows swapping clothes and virtual try-on.