Robust 3D Garment Digitization from Monocular 2D Images for 3D Virtual Try-On Systems

TL;DR

This paper presents a robust method for converting 2D images of garments into accurate 3D models with textures, enabling realistic virtual try-on applications, even with occlusions and pose variations.

Contribution

The authors introduce a novel pipeline combining landmark prediction, Thin-Plate-Spline texture transfer, and deep inpainting to generate detailed 3D garment textures from monocular images.

Findings

High-quality texture transfer on real-world images

Effective handling of occlusions and pose variations

Impressive qualitative results on fashion catalog images

Abstract

In this paper, we develop a robust 3D garment digitization solution that can generalize well on real-world fashion catalog images with cloth texture occlusions and large body pose variations. We assumed fixed topology parametric template mesh models for known types of garments (e.g., T-shirts, Trousers) and perform mapping of high-quality texture from an input catalog image to UV map panels corresponding to the parametric mesh model of the garment. We achieve this by first predicting a sparse set of 2D landmarks on the boundary of the garments. Subsequently, we use these landmarks to perform Thin-Plate-Spline-based texture transfer on UV map panels. Subsequently, we employ a deep texture inpainting network to fill the large holes (due to view variations & self-occlusions) in TPS output to generate consistent UV maps. Furthermore, to train the supervised deep networks for landmark prediction & texture inpainting tasks, we generated a large set of synthetic data with varying texture and lighting imaged from various views with the human present in a wide variety of poses. Additionally, we manually annotated a small set of fashion catalog images crawled from online fashion e-commerce platforms to finetune. We conduct thorough empirical evaluations and show impressive qualitative results of our proposed 3D garment texture solution on fashion catalog images. Such 3D garment digitization helps us solve the challenging task of enabling 3D Virtual Try-on.