Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances

TL;DR

This paper introduces a versatile robot manipulation system that integrates perception modules with task-and-motion planning, enabling the robot to perform complex tasks with unknown objects without prior knowledge or retraining.

Contribution

It presents a novel integrated framework combining general-purpose planning with learned perception modules for unknown object manipulation.

Findings

Successfully performed diverse multi-step manipulation tasks

Generalized across various objects and environments

Operated without prior environment knowledge or retraining

Abstract

We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that following this strategy a task-and-motion planner can be used to plan intelligent behaviors even in the absence of a priori knowledge regarding the set of manipulable objects, their geometries, and their affordances. We explore several different ways of implementing such perceptual modules for segmentation, property detection, shape estimation, and grasp generation. We show how these modules are integrated within the PDDLStream task and motion planning framework. Finally, we demonstrate that this strategy can enable a single system to perform a wide variety of real-world multi-step manipulation tasks, generalizing over a broad class of objects, object arrangements, and goals, without any prior knowledge of the environment and without re-training.