DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy

TL;DR

DexGarmentLab introduces a realistic simulation environment for dexterous garment manipulation, along with a novel hierarchical policy that generalizes across diverse garment types and deformations, advancing robotic handling of clothing tasks.

Contribution

The paper presents DexGarmentLab, a new environment for bimanual garment manipulation, and proposes HALO, a hierarchical policy that improves generalization to unseen garments and complex deformations.

Findings

HALO outperforms existing methods in generalization

Successfully handles unseen garment shapes and deformations

Reduces manual data collection through automatic dataset generation

Abstract

Garment manipulation is a critical challenge due to the diversity in garment categories, geometries, and deformations. Despite this, humans can effortlessly handle garments, thanks to the dexterity of our hands. However, existing research in the field has struggled to replicate this level of dexterity, primarily hindered by the lack of realistic simulations of dexterous garment manipulation. Therefore, we propose DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation, which features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap. Previous data collection typically relies on teleoperation or training expert reinforcement learning (RL) policies, which are labor-intensive and inefficient. In this paper, we leverage garment structural correspondence to automatically generate a dataset with diverse trajectories using only a single expert demonstration, significantly reducing manual intervention. However, even extensive demonstrations cannot cover the infinite states of garments, which necessitates the exploration of new algorithms. To improve generalization across diverse garment shapes and deformations, we propose a Hierarchical gArment-manipuLation pOlicy (HALO). It first identifies transferable affordance points to accurately locate the manipulation area, then generates generalizable trajectories to complete the task. Through extensive experiments and detailed analysis of our method and baseline, we demonstrate that HALO consistently outperforms existing methods, successfully generalizing to previously unseen instances even with significant variations in shape and deformation where others fail. Our project page is available at: https://wayrise.github.io/DexGarmentLab/.