Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers

TL;DR

This paper introduces a graph neural network framework that predicts how multiple objects' relations change during manipulation, enabling robots to plan multi-step interactions with variable objects in real-world scenarios.

Contribution

A novel GNN-based model for dynamic multi-object relation reasoning and planning, trained in simulation and transferred to real-world robotic manipulation tasks.

Findings

Model accurately predicts inter-object relation changes.

Enables multi-step planning for object rearrangement.

Successfully transfers from simulation to real-world applications.

Abstract

Objects rarely sit in isolation in human environments. As such, we'd like our robots to reason about how multiple objects relate to one another and how those relations may change as the robot interacts with the world. To this end, we propose a novel graph neural network framework for multi-object manipulation to predict how inter-object relations change given robot actions. Our model operates on partial-view point clouds and can reason about multiple objects dynamically interacting during the manipulation. By learning a dynamics model in a learned latent graph embedding space, our model enables multi-step planning to reach target goal relations. We show our model trained purely in simulation transfers well to the real world. Our planner enables the robot to rearrange a variable number of objects with a range of shapes and sizes using both push and pick and place skills.