Learning Visuotactile Skills with Two Multifingered Hands
Toru Lin, Yu Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine,, Jitendra Malik

TL;DR
This paper introduces HATO, a low-cost teleoperation system and hardware adaptation for multifingered hands with touch sensors, enabling learning of complex bimanual manipulation skills from visuotactile data.
Contribution
The paper presents HATO, an affordable teleoperation system and a novel hardware setup with touch sensors, facilitating research in visuotactile learning for bimanual dexterity.
Findings
HATO enables efficient data collection for visuotactile learning.
Visuotactile data improves high-precision bimanual task performance.
Dataset size and sensing modalities significantly impact policy learning.
Abstract
Aiming to replicate human-like dexterity, perceptual experiences, and motion patterns, we explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Two significant challenges exist: the lack of an affordable and accessible teleoperation system suitable for a dual-arm setup with multifingered hands, and the scarcity of multifingered hand hardware equipped with touch sensing. To tackle the first challenge, we develop HATO, a low-cost hands-arms teleoperation system that leverages off-the-shelf electronics, complemented with a software suite that enables efficient data collection; the comprehensive software suite also supports multimodal data processing, scalable policy learning, and smooth policy deployment. To tackle the latter challenge, we introduce a novel hardware adaptation by repurposing two prosthetic hands equipped with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTactile and Sensory Interactions · Ergonomics and Musculoskeletal Disorders · Virtual Reality Applications and Impacts
