RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing

TL;DR

RoboPack introduces a neural tactile-informed dynamics model combining visual and tactile data, enabling robots to perform dense packing and manipulation tasks with limited real-world training data.

Contribution

The paper presents RoboPack, a novel framework that integrates tactile feedback into dynamics modeling using a recurrent graph neural network for improved manipulation.

Findings

Effective in dense packing and manipulation tasks.

Achieves online adaptation with only 30 minutes of data.

Outperforms previous learning-based and physics-based methods.

Abstract

Tactile feedback is critical for understanding the dynamics of both rigid and deformable objects in many manipulation tasks, such as non-prehensile manipulation and dense packing. We introduce an approach that combines visual and tactile sensing for robotic manipulation by learning a neural, tactile-informed dynamics model. Our proposed framework, RoboPack, employs a recurrent graph neural network to estimate object states, including particles and object-level latent physics information, from historical visuo-tactile observations and to perform future state predictions. Our tactile-informed dynamics model, learned from real-world data, can solve downstream robotics tasks with model-predictive control. We demonstrate our approach on a real robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile manipulation and dense packing tasks, where the robot must infer the physics properties of objects from direct and indirect interactions. Trained on only an average of 30 minutes of real-world interaction data per task, our model can perform online adaptation and make touch-informed predictions. Through extensive evaluations in both long-horizon dynamics prediction and real-world manipulation, our method demonstrates superior effectiveness compared to previous learning-based and physics-based simulation systems.