GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoning

TL;DR

This paper introduces GarmentPile++, a novel vision-language reasoning system that effectively retrieves individual garments from cluttered piles for home-assistant robotics, integrating segmentation, affordance perception, and dual-arm cooperation.

Contribution

It presents a new retrieval pipeline combining high-level reasoning with visual affordance perception, enhanced by segmentation and a dual-arm framework for complex garment handling.

Findings

Effective garment retrieval in cluttered piles demonstrated in real-world and simulation environments.

Robust segmentation and reasoning improve retrieval accuracy and safety.

Dual-arm cooperation handles large or sagging garments efficiently.

Abstract

Garment manipulation has attracted increasing attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment retrieval pipeline that can not only follow language instruction to execute safe and clean retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates vision-language reasoning with visual affordance perception, fully leveraging the high-level reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM's comprehensive awareness of each garment's state within a garment pile, we employ visual segmentation model (SAM2) to execute object segmentation on the garment pile for aiding VLM-based reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/.