Visuotactile-Based Learning for Insertion with Compliant Hands

TL;DR

This paper presents a visuotactile sensing approach combined with deep learning to improve insertion tasks with compliant robotic hands, emphasizing robust perception and sim-to-real transfer.

Contribution

It introduces a multimodal policy learning framework using tactile and visual sensing, with a transformer-based policy transferred to real robots without additional training.

Findings

Tactile sensing enhances object pose estimation accuracy.

The proposed method achieves successful sim-to-real transfer.

Robust insertion performance is demonstrated in real-world tests.

Abstract

Compared to rigid hands, underactuated compliant hands offer greater adaptability to object shapes, provide stable grasps, and are often more cost-effective. However, they introduce uncertainties in hand-object interactions due to their inherent compliance and lack of precise finger proprioception as in rigid hands. These limitations become particularly significant when performing contact-rich tasks like insertion. To address these challenges, additional sensing modalities are required to enable robust insertion capabilities. This letter explores the essential sensing requirements for successful insertion tasks with compliant hands, focusing on the role of visuotactile perception (i.e., visual and tactile perception). We propose a simulation-based multimodal policy learning framework that leverages all-around tactile sensing and an extrinsic depth camera. A transformer-based policy, trained through a teacher-student distillation process, is successfully transferred to a real-world robotic system without further training. Our results emphasize the crucial role of tactile sensing in conjunction with visual perception for accurate object-socket pose estimation, successful sim-to-real transfer and robust task execution.