Learning Robust Dexterous In-Hand Manipulation from Joint Sensors with Proprioceptive Transformer

TL;DR

This paper introduces Proprioceptive Transformer, a novel approach enabling dexterous in-hand object manipulation using only joint sensors, achieving high rotation speed and accurate object state estimation without external perception.

Contribution

The paper presents a transformer-based method that extracts environment information solely from joint sensor data, enabling effective in-hand manipulation without vision or tactile sensing.

Findings

Achieves 3.1x higher rotation speed than baselines.

Reduces RMSE for cube position estimation by 23.4%.

Demonstrates effective proprioceptive-only control on real robotic hand.

Abstract

In-hand object manipulation is a fundamental yet challenging capability for dexterous robots. Despite significant progress in dexterous manipulation, existing approaches rely heavily on vision or tactile sensing to track object states, while joint sensing -- the most readily available modality on any robotic hand -- remains largely overlooked, particularly for tendon-driven hands. In this paper, we study how far joint sensing alone can go by asking: (i) whether motor encoders or direct joint sensing provides better proprioceptive feedback, (ii) how to extract environment information from joint measurements, and (iii) whether joint-only control can achieve competitive real-world performance without external perception. We present the Proprioceptive Transformer (PT), an exteroceptive-free approach for continuous cube rotation on a tendon-driven dexterous hand that uses only joint sensing feedback. A teacher policy is first trained via reinforcement learning with privileged object information, then distilled into PT, which operates solely on joint position and velocity histories. The Transformer architecture effectively extracts implicit object state information from temporal patterns in joint sensor readings. Experiments on the real ORCA hand show that our approach achieves 3.1x higher rotation speed than baselines. We also demonstrate that our PT achieves a 23.4% lower RMSE for cube position estimation than the MLP baseline, indicating superior extraction of exteroceptive information from proprioceptive sources.