Ab Initio Particle-based Object Manipulation

TL;DR

This paper introduces Particle-based Object Manipulation (Prompt), a novel real-time approach that combines model-based reasoning and data-driven learning for robot manipulation of novel objects without prior models or training.

Contribution

Prompt is the first particle-based manipulation method that learns online from visual input, enabling effective manipulation of diverse objects without offline training.

Findings

Successfully handles various everyday objects, including transparent ones.

Outperforms state-of-the-art data-driven grasping methods.

Operates in real-time without prior object models or offline training.

Abstract

This paper presents Particle-based Object Manipulation (Prompt), a new approach to robot manipulation of novel objects ab initio, without prior object models or pre-training on a large object data set. The key element of Prompt is a particle-based object representation, in which each particle represents a point in the object, the local geometric, physical, and other features of the point, and also its relation with other particles. Like the model-based analytic approaches to manipulation, the particle representation enables the robot to reason about the object's geometry and dynamics in order to choose suitable manipulation actions. Like the data-driven approaches, the particle representation is learned online in real-time from visual sensor input, specifically, multi-view RGB images. The particle representation thus connects visual perception with robot control. Prompt combines the benefits of both model-based reasoning and data-driven learning. We show empirically that Prompt successfully handles a variety of everyday objects, some of which are transparent. It handles various manipulation tasks, including grasping, pushing, etc,. Our experiments also show that Prompt outperforms a state-of-the-art data-driven grasping method on the daily objects, even though it does not use any offline training data.