DexHandDiff: Interaction-aware Diffusion Planning for Adaptive Dexterous Manipulation

TL;DR

DexHandDiff is a novel diffusion-based planning framework that enables adaptive, goal-oriented dexterous manipulation in robotics, effectively handling complex contact-rich tasks with high success rates.

Contribution

It introduces an interaction-aware diffusion process with dual guidance and language model integration for improved adaptability and realism in dexterous manipulation planning.

Findings

Over twice the success rate compared to existing methods.

Achieves 70.7% success on goal-adaptive tasks.

Effective on diverse physical interaction tasks.

Abstract

Dexterous manipulation with contact-rich interactions is crucial for advanced robotics. While recent diffusion-based planning approaches show promise for simple manipulation tasks, they often produce unrealistic ghost states (e.g., the object automatically moves without hand contact) or lack adaptability when handling complex sequential interactions. In this work, we introduce DexHandDiff, an interaction-aware diffusion planning framework for adaptive dexterous manipulation. DexHandDiff models joint state-action dynamics through a dual-phase diffusion process which consists of pre-interaction contact alignment and post-contact goal-directed control, enabling goal-adaptive generalizable dexterous manipulation. Additionally, we incorporate dynamics model-based dual guidance and leverage large language models for automated guidance function generation, enhancing generalizability for physical interactions and facilitating diverse goal adaptation through language cues. Experiments on physical interaction tasks such as door opening, pen and block re-orientation, object relocation, and hammer striking demonstrate DexHandDiff's effectiveness on goals outside training distributions, achieving over twice the average success rate (59.2% vs. 29.5%) compared to existing methods. Our framework achieves an average of 70.7% success rate on goal adaptive dexterous tasks, highlighting its robustness and flexibility in contact-rich manipulation.