A Dataset and Benchmark for Robotic Cloth Unfolding Grasp Selection: The ICRA 2024 Cloth Competition

TL;DR

This paper introduces a new benchmark, dataset, and competition for evaluating robotic cloth unfolding methods, highlighting the importance of standardized evaluation and providing resources for future research in data-driven cloth manipulation.

Contribution

The paper presents the ICRA 2024 Cloth Competition, a comprehensive dataset, and analysis of grasp selection approaches, establishing a standardized benchmark for robotic cloth unfolding evaluation.

Findings

Hand-engineered methods perform surprisingly well.

Significant gap between competition results and prior work.

Insights into trade-offs between grasp success and coverage.

Abstract

Robotic cloth manipulation suffers from a lack of standardized benchmarks and shared datasets for evaluating and comparing different approaches. To address this, we created a benchmark and organized the ICRA 2024 Cloth Competition, a unique head-to-head evaluation focused on grasp pose selection for in-air robotic cloth unfolding. Eleven diverse teams participated in the competition, utilizing our publicly released dataset of real-world robotic cloth unfolding attempts and a variety of methods to design their unfolding approaches. Afterwards, we also expanded our dataset with 176 competition evaluation trials, resulting in a dataset of 679 unfolding demonstrations across 34 garments. Analysis of the competition results revealed insights about the trade-off between grasp success and coverage, the surprisingly strong achievements of hand-engineered methods and a significant discrepancy between competition performance and prior work, underscoring the importance of independent, out-of-the-lab evaluation in robotic cloth manipulation. The associated dataset is a valuable resource for developing and evaluating grasp selection methods, particularly for learning-based approaches. We hope that our benchmark, dataset and competition results can serve as a foundation for future benchmarks and drive further progress in data-driven robotic cloth manipulation. The dataset and benchmarking code are available at https://airo.ugent.be/cloth_competition.