DextrAH-RGB: Visuomotor Policies to Grasp Anything with Dexterous Hands

TL;DR

DextrAH-RGB is a novel system enabling dexterous robot grasping directly from RGB images, demonstrating robust sim2real transfer and generalization to unseen objects without relying on depth data.

Contribution

The paper introduces the first end-to-end RGB-based dexterous grasping policy with successful sim2real transfer and broad object generalization.

Findings

Achieves robust sim2real transfer for complex grasping tasks

Generalizes to novel objects with different textures and lighting

Performs comparably to depth-based grasping methods

Abstract

One of the most important, yet challenging, skills for a dexterous robot is grasping a diverse range of objects. Much of the prior work has been limited by speed, generality, or reliance on depth maps and object poses. In this paper, we introduce DextrAH-RGB, a system that can perform dexterous arm-hand grasping end-to-end from RGB image input. We train a privileged fabric-guided policy (FGP) in simulation through reinforcement learning that acts on a geometric fabric controller to dexterously grasp a wide variety of objects. We then distill this privileged FGP into a RGB-based FGP strictly in simulation using photorealistic tiled rendering. To our knowledge, this is the first work that is able to demonstrate robust sim2real transfer of an end2end RGB-based policy for complex, dynamic, contact-rich tasks such as dexterous grasping. DextrAH-RGB is competitive with depth-based dexterous grasping policies, and generalizes to novel objects with unseen geometry, texture, and lighting conditions in the real world. Videos of our system grasping a diverse range of unseen objects are available at \url{https://dextrah-rgb.github.io/}.