Leveraging Pretrained Latent Representations for Few-Shot Imitation Learning on a Dexterous Robotic Hand

TL;DR

This paper introduces a method that leverages large-scale human hand datasets to improve few-shot imitation learning on a dexterous robotic hand, enhancing robustness and eliminating the need for teleoperation.

Contribution

It proposes a novel approach using latent representations from multiple datasets within a transformer-based behavior cloning framework for robotic manipulation.

Findings

Enhanced performance over traditional behavior cloning methods.

Increased resilience to perception and proprioception noise.

Successful transfer of policies to a real-world 23-DoF robotic hand.

Abstract

In the context of imitation learning applied to dexterous robotic hands, the high complexity of the systems makes learning complex manipulation tasks challenging. However, the numerous datasets depicting human hands in various different tasks could provide us with better knowledge regarding human hand motion. We propose a method to leverage multiple large-scale task-agnostic datasets to obtain latent representations that effectively encode motion subtrajectories that we included in a transformer-based behavior cloning method. Our results demonstrate that employing latent representations yields enhanced performance compared to conventional behavior cloning methods, particularly regarding resilience to errors and noise in perception and proprioception. Furthermore, the proposed approach solely relies on human demonstrations, eliminating the need for teleoperation and, therefore, accelerating the data acquisition process. Accurate inverse kinematics for fingertip retargeting ensures precise transfer from human hand data to the robot, facilitating effective learning and deployment of manipulation policies. Finally, the trained policies have been successfully transferred to a real-world 23Dof robotic system.