Self-supervised perception for tactile skin covered dexterous hands

TL;DR

This paper introduces Sparsh-skin, a self-supervised encoder for magnetic tactile sensors on robotic hands, enabling improved perception and task performance across various applications.

Contribution

It presents a novel self-supervised pre-trained model for magnetic tactile sensors, enhancing general-purpose tactile perception for dexterous robotic hands.

Findings

Pretrained Sparsh-skin improves downstream task performance by over 41%.

Sparsh-skin is sample efficient in learning new tasks.

It outperforms prior methods and end-to-end learning significantly.

Abstract

We present Sparsh-skin, a pre-trained encoder for magnetic skin sensors distributed across the fingertips, phalanges, and palm of a dexterous robot hand. Magnetic tactile skins offer a flexible form factor for hand-wide coverage with fast response times, in contrast to vision-based tactile sensors that are restricted to the fingertips and limited by bandwidth. Full hand tactile perception is crucial for robot dexterity. However, a lack of general-purpose models, challenges with interpreting magnetic flux and calibration have limited the adoption of these sensors. Sparsh-skin, given a history of kinematic and tactile sensing across a hand, outputs a latent tactile embedding that can be used in any downstream task. The encoder is self-supervised via self-distillation on a variety of unlabeled hand-object interactions using an Allegro hand sensorized with Xela uSkin. In experiments across several benchmark tasks, from state estimation to policy learning, we find that pretrained Sparsh-skin representations are both sample efficient in learning downstream tasks and improve task performance by over 41% compared to prior work and over 56% compared to end-to-end learning.