TransDex: Pre-training Visuo-Tactile Policy with Point Cloud Reconstruction for Dexterous Manipulation of Transparent Objects

TL;DR

TransDex introduces a pre-trained 3D visuo-tactile policy using point cloud reconstruction with Transformers, significantly improving transparent object manipulation in robotics.

Contribution

The paper presents a novel self-supervised point cloud reconstruction pre-training approach and a hierarchical multi-round attention mechanism for visuo-tactile fusion in dexterous manipulation.

Findings

TransDex outperforms baseline methods in transparent object manipulation.

The approach demonstrates strong generalization capabilities.

Component analysis confirms effectiveness of the proposed modules.

Abstract

Dexterous manipulation enables complex tasks but suffers from self-occlusion, severe depth noise, and depth information loss when manipulating transparent objects. To solve this problem, this paper proposes TransDex, a 3D visuo-tactile fusion motor policy based on point cloud reconstruction pre-training. Specifically, we first propose a self-supervised point cloud reconstruction pre-training approach based on Transformer. This method accurately recovers the 3D structure of objects from interactive point clouds of dexterous hands, even when random noise and large-scale masking are added. Building on this, TransDex is constructed in which perceptual encoding adopts a fine-grained hierarchical scheme and multi-round attention mechanisms adaptively fuse features of the robotic arm and dexterous hand to enable differentiated motion prediction. Results from transparent object manipulation experiments conducted on a real robotic system demonstrate that TransDex outperforms existing baseline methods. Further analysis validates the generalization capabilities of TransDex and the effectiveness of its individual components.