Neural 3D Clothes Retargeting from a Single Image

TL;DR

This paper introduces a neural network-based method for 3D clothes retargeting from a single RGB image, leveraging synthetic data and semi-supervised learning to predict realistic clothing deformations and poses.

Contribution

It presents a novel semi-supervised framework and neural network (CRNet) for 3D clothes retargeting using synthetic data, addressing the ill-posed nature of the problem.

Findings

Accurately predicts 3D clothing deformation and pose from a single image.

Uses synthetic data to overcome lack of ground truth in real images.

Demonstrates realistic clothing retargeting in real-world scenarios.

Abstract

In this paper, we present a method of clothes retargeting; generating the potential poses and deformations of a given 3D clothing template model to fit onto a person in a single RGB image. The problem is fundamentally ill-posed as attaining the ground truth data is impossible, i.e., images of people wearing the different 3D clothing template model at exact same pose. We address this challenge by utilizing large-scale synthetic data generated from physical simulation, allowing us to map 2D dense body pose to 3D clothing deformation. With the simulated data, we propose a semi-supervised learning framework that validates the physical plausibility of the 3D deformation by matching with the prescribed body-to-cloth contact points and clothing silhouette to fit onto the unlabeled real images. A new neural clothes retargeting network (CRNet) is designed to integrate the semi-supervised retargeting task in an end-to-end fashion. In our evaluation, we show that our method can predict the realistic 3D pose and deformation field needed for retargeting clothes models in real-world examples.