Flat'n'Fold: A Diverse Multi-Modal Dataset for Garment Perception and Manipulation

TL;DR

Flat'n'Fold is a comprehensive large-scale dataset capturing diverse human and robot garment manipulation tasks, including multi-view RGB-D data, to advance perception and manipulation of deformable objects.

Contribution

It introduces a novel, large, and diverse dataset for garment manipulation, covering the entire process with synchronized multi-modal data, surpassing existing datasets in scope and detail.

Findings

State-of-the-art models show significant room for improvement on new benchmarks.

The dataset enables evaluation of grasping point prediction and subtask decomposition.

Demonstrates the diversity and complexity of real-world garment manipulation.

Abstract

We present Flat'n'Fold, a novel large-scale dataset for garment manipulation that addresses critical gaps in existing datasets. Comprising 1,212 human and 887 robot demonstrations of flattening and folding 44 unique garments across 8 categories, Flat'n'Fold surpasses prior datasets in size, scope, and diversity. Our dataset uniquely captures the entire manipulation process from crumpled to folded states, providing synchronized multi-view RGB-D images, point clouds, and action data, including hand or gripper positions and rotations. We quantify the dataset's diversity and complexity compared to existing benchmarks and show that our dataset features natural and diverse manipulations of real-world demonstrations of human and robot demonstrations in terms of visual and action information. To showcase Flat'n'Fold's utility, we establish new benchmarks for grasping point prediction and subtask decomposition. Our evaluation of state-of-the-art models on these tasks reveals significant room for improvement. This underscores Flat'n'Fold's potential to drive advances in robotic perception and manipulation of deformable objects. Our dataset can be downloaded at https://cvas-ug.github.io/flat-n-fold