Self-supervised Cloth Reconstruction via Action-conditioned Cloth Tracking

TL;DR

This paper introduces a self-supervised approach to improve cloth mesh reconstruction in real-world scenarios by using action-conditioned tracking and pseudo-labels, addressing the sim-to-real gap in cloth manipulation tasks.

Contribution

It presents a novel self-supervised finetuning method for cloth mesh reconstruction that leverages action-conditioned tracking and pseudo-labels without human annotations.

Findings

Enhanced mesh reconstruction quality in real-world settings

Improved downstream cloth manipulation performance

Effective self-supervised finetuning method

Abstract

State estimation is one of the greatest challenges for cloth manipulation due to cloth's high dimensionality and self-occlusion. Prior works propose to identify the full state of crumpled clothes by training a mesh reconstruction model in simulation. However, such models are prone to suffer from a sim-to-real gap due to differences between cloth simulation and the real world. In this work, we propose a self-supervised method to finetune a mesh reconstruction model in the real world. Since the full mesh of crumpled cloth is difficult to obtain in the real world, we design a special data collection scheme and an action-conditioned model-based cloth tracking method to generate pseudo-labels for self-supervised learning. By finetuning the pretrained mesh reconstruction model on this pseudo-labeled dataset, we show that we can improve the quality of the reconstructed mesh without requiring human annotations, and improve the performance of downstream manipulation task.