RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

TL;DR

RealDex is a comprehensive dataset of human-like dexterous hand grasping motions, combined with a novel motion generation framework that leverages multimodal large language models to improve robotic grasping capabilities.

Contribution

The paper introduces the first dataset of authentic human-like dexterous hand motions and a new motion generation framework utilizing multimodal large language models for robotic grasping.

Findings

Our method outperforms existing approaches on RealDex and other datasets.

The dataset enables more natural and precise human-like robotic grasping.

The framework enhances real-world robotic manipulation tasks.

Abstract

In this paper, we introduce RealDex, a pioneering dataset capturing authentic dexterous hand grasping motions infused with human behavioral patterns, enriched by multi-view and multimodal visual data. Utilizing a teleoperation system, we seamlessly synchronize human-robot hand poses in real time. This collection of human-like motions is crucial for training dexterous hands to mimic human movements more naturally and precisely. RealDex holds immense promise in advancing humanoid robot for automated perception, cognition, and manipulation in real-world scenarios. Moreover, we introduce a cutting-edge dexterous grasping motion generation framework, which aligns with human experience and enhances real-world applicability through effectively utilizing Multimodal Large Language Models. Extensive experiments have demonstrated the superior performance of our method on RealDex and other open datasets. The complete dataset and code will be made available upon the publication of this work.