ClutterDexGrasp: A Sim-to-Real System for General Dexterous Grasping in Cluttered Scenes

TL;DR

This paper introduces ClutterDexGrasp, a zero-shot sim-to-real system for general dexterous grasping in cluttered scenes, combining simulation-trained policies with real-world deployment without additional training.

Contribution

It presents a novel two-stage teacher-student framework with curriculum learning and a 3D diffusion policy for robust, zero-shot transfer in complex cluttered environments.

Findings

Achieves zero-shot sim-to-real transfer for dexterous grasping

Demonstrates robust performance across diverse objects and clutter layouts

Introduces a safety curriculum for collision-free grasping

Abstract

Dexterous grasping in cluttered scenes presents significant challenges due to diverse object geometries, occlusions, and potential collisions. Existing methods primarily focus on single-object grasping or grasp-pose prediction without interaction, which are insufficient for complex, cluttered scenes. Recent vision-language-action models offer a potential solution but require extensive real-world demonstrations, making them costly and difficult to scale. To address these limitations, we revisit the sim-to-real transfer pipeline and develop key techniques that enable zero-shot deployment in reality while maintaining robust generalization. We propose ClutterDexGrasp, a two-stage teacher-student framework for closed-loop target-oriented dexterous grasping in cluttered scenes. The framework features a teacher policy trained in simulation using clutter density curriculum learning, incorporating both a geometry and spatially-embedded scene representation and a novel comprehensive safety curriculum, enabling general, dynamic, and safe grasping behaviors. Through imitation learning, we distill the teacher's knowledge into a student 3D diffusion policy (DP3) that operates on partial point cloud observations. To the best of our knowledge, this represents the first zero-shot sim-to-real closed-loop system for target-oriented dexterous grasping in cluttered scenes, demonstrating robust performance across diverse objects and layouts. More details and videos are available at https://clutterdexgrasp.github.io/.