Universal Dexterous Functional Grasping via Demonstration-Editing Reinforcement Learning

TL;DR

This paper introduces DemoFunGrasp, a reinforcement learning framework for universal dexterous functional grasping that leverages demonstration editing and vision-language models to improve generalization, efficiency, and real-world applicability.

Contribution

It proposes a novel RL approach that factorizes grasping conditions and uses demonstration editing, enhancing multi-task learning and sim-to-real transfer for functional grasping.

Findings

Outperforms baselines in success rate and grasping accuracy

Demonstrates strong sim-to-real transfer capabilities

Enables autonomous instruction-following grasp execution

Abstract

Reinforcement learning (RL) has achieved great success in dexterous grasping, significantly improving grasp performance and generalization from simulation to the real world. However, fine-grained functional grasping, which is essential for downstream manipulation tasks, remains underexplored and faces several challenges: the complexity of specifying goals and reward functions for functional grasps across diverse objects, the difficulty of multi-task RL exploration, and the challenge of sim-to-real transfer. In this work, we propose DemoFunGrasp for universal dexterous functional grasping. We factorize functional grasping conditions into two complementary components - grasping style and affordance - and integrate them into an RL framework that can learn to grasp any object with any functional grasping condition. To address the multi-task optimization challenge, we leverage a single grasping demonstration and reformulate the RL problem as one-step demonstration editing, substantially enhancing sample efficiency and performance. Experimental results in both simulation and the real world show that DemoFunGrasp generalizes to unseen combinations of objects, affordances, and grasping styles, outperforming baselines in both success rate and functional grasping accuracy. In addition to strong sim-to-real capability, by incorporating a vision-language model (VLM) for planning, our system achieves autonomous instruction-following grasp execution.