OmniDexGrasp: Generalizable Dexterous Grasping via Foundation Model and Force Feedback

TL;DR

OmniDexGrasp introduces a framework combining foundation models and force feedback to enable robots to perform generalizable, dexterous grasping and manipulation based on human commands, demonstrating robustness across diverse objects and tasks.

Contribution

The paper presents OmniDexGrasp, a novel framework that integrates foundation models with transfer and control strategies for generalizable dexterous robotic grasping and manipulation.

Findings

Effective in simulation and real robot experiments

Handles diverse prompts and grasp tasks

Extensible to dexterous manipulation

Abstract

Enabling robots to dexterously grasp and manipulate objects based on human commands is a promising direction in robotics. However, existing approaches are challenging to generalize across diverse objects or tasks due to the limited scale of semantic dexterous grasp datasets. Foundation models offer a new way to enhance generalization, yet directly leveraging them to generate feasible robotic actions remains challenging due to the gap between abstract model knowledge and physical robot execution. To address these challenges, we propose OmniDexGrasp, a generalizable framework that achieves omni-capabilities in user prompting, dexterous embodiment, and grasping tasks by combining foundation models with the transfer and control strategies. OmniDexGrasp integrates three key modules: (i) foundation models are used to enhance generalization by generating human grasp images supporting omni-capability of user prompt and task; (ii) a human-image-to-robot-action transfer strategy converts human demonstrations into executable robot actions, enabling omni dexterous embodiment; (iii) force-aware adaptive grasp strategy ensures robust and stable grasp execution. Experiments in simulation and on real robots validate the effectiveness of OmniDexGrasp on diverse user prompts, grasp task and dexterous hands, and further results show its extensibility to dexterous manipulation tasks.