Adversarial Object Rearrangement in Constrained Environments with Heterogeneous Graph Neural Networks

TL;DR

This paper introduces a hierarchical robotic system utilizing heterogeneous graph neural networks to understand and manipulate objects in constrained environments, demonstrating high success rates in real-world tests.

Contribution

The novel integration of HetGNN for reasoning about scene components and a hierarchical control system for adversarial object rearrangement in constrained settings.

Findings

Achieved 87.88% success rate in real-world experiments.

Surpassed baseline methods in rearrangement tasks.

Effective simulation-to-real transfer demonstrated.

Abstract

Adversarial object rearrangement in the real world (e.g., previously unseen or oversized items in kitchens and stores) could benefit from understanding task scenes, which inherently entail heterogeneous components such as current objects, goal objects, and environmental constraints. The semantic relationships among these components are distinct from each other and crucial for multi-skilled robots to perform efficiently in everyday scenarios. We propose a hierarchical robotic manipulation system that learns the underlying relationships and maximizes the collaborative power of its diverse skills (e.g., pick-place, push) for rearranging adversarial objects in constrained environments. The high-level coordinator employs a heterogeneous graph neural network (HetGNN), which reasons about the current objects, goal objects, and environmental constraints; the low-level 3D Convolutional Neural Network-based actors execute the action primitives. Our approach is trained entirely in simulation, and achieved an average success rate of 87.88% and a planning cost of 12.82 in real-world experiments, surpassing all baseline methods. Supplementary material is available at https://sites.google.com/umn.edu/versatile-rearrangement.