TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot

TL;DR

This paper introduces TacGNN, a graph-based tactile perception model enabling a blind robotic hand to learn in-hand manipulation tasks solely through tactile sensing, demonstrating effective object state prediction and successful task transfer to real robots.

Contribution

The paper presents a novel tactile perception framework using TacGNN for in-hand manipulation without visual input, advancing tactile-based robotic dexterity learning.

Findings

TacGNN reduces prediction RMSE to 0.096cm, outperforming other models.

The robot successfully completes manipulation tasks using only tactile and proprioceptive data.

Methods transfer effectively from simulation to real robot without retraining.

Abstract

In this paper, we propose a novel framework for tactile-based dexterous manipulation learning with a blind anthropomorphic robotic hand, i.e. without visual sensing. First, object-related states were extracted from the raw tactile signals by a graph-based perception model - TacGNN. The resulting tactile features were then utilized in the policy learning of an in-hand manipulation task in the second stage. This method was examined by a Baoding ball task - simultaneously manipulating two spheres around each other by 180 degrees in hand. We conducted experiments on object states prediction and in-hand manipulation using a reinforcement learning algorithm (PPO). Results show that TacGNN is effective in predicting object-related states during manipulation by decreasing the RMSE of prediction to 0.096cm comparing to other methods, such as MLP, CNN, and GCN. Finally, the robot hand could finish an in-hand manipulation task solely relying on the robotic own perception - tactile sensing and proprioception. In addition, our methods are tested on three tasks with different difficulty levels and transferred to the real robot without further training.