Foldsformer: Learning Sequential Multi-Step Cloth Manipulation With Space-Time Attention

TL;DR

Foldformer is a novel framework that enables robots to perform multi-step cloth manipulation tasks using space-time attention, general demonstrations, and transferability to real-world and unseen cloth shapes without additional training.

Contribution

The paper introduces Foldformer, a new multi-step cloth manipulation framework that leverages space-time attention and general demonstrations for improved flexibility and transferability.

Findings

Outperforms state-of-the-art in simulation on four tasks.

Successfully transfers from simulation to real world without extra training.

Generalizes to unseen cloth shapes like T-shirts and shorts.

Abstract

Sequential multi-step cloth manipulation is a challenging problem in robotic manipulation, requiring a robot to perceive the cloth state and plan a sequence of chained actions leading to the desired state. Most previous works address this problem in a goal-conditioned way, and goal observation must be given for each specific task and cloth configuration, which is not practical and efficient. Thus, we present a novel multi-step cloth manipulation planning framework named Foldformer. Foldformer can complete similar tasks with only a general demonstration and utilize a space-time attention mechanism to capture the instruction information behind this demonstration. We experimentally evaluate Foldsformer on four representative sequential multi-step manipulation tasks and show that Foldsformer significantly outperforms state-of-the-art approaches in simulation. Foldformer can complete multi-step cloth manipulation tasks even when configurations of the cloth (e.g., size and pose) vary from configurations in the general demonstrations. Furthermore, our approach can be transferred from simulation to the real world without additional training or domain randomization. Despite training on rectangular clothes, we also show that our approach can generalize to unseen cloth shapes (T-shirts and shorts). Videos and source code are available at: https://sites.google.com/view/foldsformer.