In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation

TL;DR

This paper introduces a real-time GPU-accelerated method for tracking in-hand object poses during manipulation by leveraging contact feedback, physics simulation, and derivative-free optimization, overcoming occlusion challenges.

Contribution

It presents a novel real-time approach combining GPU simulation and sample-based optimization for dynamic in-hand object pose tracking, extending prior static grasp methods.

Findings

Achieves 6mm average point cloud error in simulation.

Achieves 13mm average point cloud error in real-world experiments.

Runs at 30Hz on a single GPU.

Abstract

Tracking the pose of an object while it is being held and manipulated by a robot hand is difficult for vision-based methods due to significant occlusions. Prior works have explored using contact feedback and particle filters to localize in-hand objects. However, they have mostly focused on the static grasp setting and not when the object is in motion, as doing so requires modeling of complex contact dynamics. In this work, we propose using GPU-accelerated parallel robot simulations and derivative-free, sample-based optimizers to track in-hand object poses with contact feedback during manipulation. We use physics simulation as the forward model for robot-object interactions, and the algorithm jointly optimizes for the state and the parameters of the simulations, so they better match with those of the real world. Our method runs in real-time (30Hz) on a single GPU, and it achieves an average point cloud distance error of 6mm in simulation experiments and 13mm in the real-world ones. View experiment videos at https://sites.google.com/view/in-hand-object-pose-tracking/