Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

TL;DR

Holo-Dex introduces an immersive mixed reality framework enabling intuitive robot skill teaching through VR, facilitating rapid collection of demonstrations and effective training of dexterous manipulation skills with generalization capabilities.

Contribution

This work presents a novel VR-based interface for dexterous robot skill teaching, combining high-fidelity hand tracking with non-parametric imitation learning, enabling rapid training and generalization.

Findings

Effective demonstration collection within hours

Successful training on six dexterous tasks

Skills generalize to unseen objects

Abstract

A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo-Dex, a framework for dexterous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-purpose dexterous tasks. Given these demonstrations, we use powerful feature learning combined with non-parametric imitation to train dexterous skills. Our experiments on six common dexterous tasks, including in-hand rotation, spinning, and bottle opening, indicate that Holo-Dex can both collect high-quality demonstration data and train skills in a matter of hours. Finally, we find that our trained skills can exhibit generalization on objects not seen in training. Videos of Holo-Dex are available at https://holo-dex.github.io.