Learning Fabric Manipulation in the Real World with Human Videos

TL;DR

This paper presents a method for learning fabric manipulation directly from human videos, enabling robots to perform tasks like folding with minimal data and no additional robot training, bridging the sim-to-real gap.

Contribution

It introduces a natural, efficient data collection pipeline from human demonstrations and demonstrates real-world fabric folding with minimal demonstrations and no robot data collection.

Findings

Robust fabric folding from crumpled states

Effective learning from few human demonstrations

No robot data collection needed for deployment

Abstract

Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising alternative is to learn fabric manipulation directly from watching humans perform the task. In this work, we explore how demonstrations for fabric manipulation tasks can be collected directly by humans, providing an extremely natural and fast data collection pipeline. Then, using only a handful of such demonstrations, we show how a pick-and-place policy can be learned and deployed on a real robot, without any robot data collection at all. We demonstrate our approach on a fabric folding task, showing that our policy can reliably reach folded states from crumpled initial configurations. Videos are available at: https://sites.google.com/view/foldingbyhand