Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation with Tools

TL;DR

This paper introduces a demonstration-free hierarchical planning method for deformable object manipulation using LLMs for high-level planning and a novel EMD-space control strategy, enabling generalization to complex long-horizon tasks without prior demonstrations.

Contribution

It presents a novel hierarchical, demonstration-free planning approach combining LLMs and EMD-space control for deformable manipulation, eliminating the need for training or demonstrations.

Findings

Outperforms benchmarks in dough manipulation tasks.

Demonstrates robust generalization to new complex tasks.

Validated on real-world robotic platforms.

Abstract

Deformable object manipulation stands as one of the most captivating yet formidable challenges in robotics. While previous techniques have predominantly relied on learning latent dynamics through demonstrations, typically represented as either particles or images, there exists a pertinent limitation: acquiring suitable demonstrations, especially for long-horizon tasks, can be elusive. Moreover, basing learning entirely on demonstrations can hamper the model's ability to generalize beyond the demonstrated tasks. In this work, we introduce a demonstration-free hierarchical planning approach capable of tackling intricate long-horizon tasks without necessitating any training. We employ large language models (LLMs) to articulate a high-level, stage-by-stage plan corresponding to a specified task. For every individual stage, the LLM provides both the tool's name and the Python code to craft intermediate subgoal point clouds. With the tool and subgoal for a particular stage at our disposal, we present a granular closed-loop model predictive control strategy. This leverages Differentiable Physics with Point-to-Point correspondence (DiffPhysics-P2P) loss in the earth mover distance (EMD) space, applied iteratively. Experimental findings affirm that our technique surpasses multiple benchmarks in dough manipulation, spanning both short and long horizons. Remarkably, our model demonstrates robust generalization capabilities to novel and previously unencountered complex tasks without any preliminary demonstrations. We further substantiate our approach with experimental trials on real-world robotic platforms. Our project page: https://qq456cvb.github.io/projects/donut.