Hierarchical Graph Neural Networks for Proprioceptive 6D Pose Estimation of In-hand Objects

TL;DR

This paper presents a hierarchical graph neural network that fuses multimodal vision and touch data, along with proprioception, to improve 6D object pose estimation in robotic in-hand manipulation, especially under occlusion.

Contribution

It introduces a novel hierarchical graph neural network architecture that effectively combines multimodal data and proprioception for accurate 6D pose estimation.

Findings

Outperforms existing methods in accuracy and robustness to occlusion

Demonstrates successful transfer from simulation to real robot

Enhances pose estimation when visual features are occluded or lacking

Abstract

Robotic manipulation, in particular in-hand object manipulation, often requires an accurate estimate of the object's 6D pose. To improve the accuracy of the estimated pose, state-of-the-art approaches in 6D object pose estimation use observational data from one or more modalities, e.g., RGB images, depth, and tactile readings. However, existing approaches make limited use of the underlying geometric structure of the object captured by these modalities, thereby, increasing their reliance on visual features. This results in poor performance when presented with objects that lack such visual features or when visual features are simply occluded. Furthermore, current approaches do not take advantage of the proprioceptive information embedded in the position of the fingers. To address these limitations, in this paper: (1) we introduce a hierarchical graph neural network architecture for combining multimodal (vision and touch) data that allows for a geometrically informed 6D object pose estimation, (2) we introduce a hierarchical message passing operation that flows the information within and across modalities to learn a graph-based object representation, and (3) we introduce a method that accounts for the proprioceptive information for in-hand object representation. We evaluate our model on a diverse subset of objects from the YCB Object and Model Set, and show that our method substantially outperforms existing state-of-the-art work in accuracy and robustness to occlusion. We also deploy our proposed framework on a real robot and qualitatively demonstrate successful transfer to real settings.