Concurrent Prehensile and Nonprehensile Manipulation: A Practical Approach to Multi-Stage Dexterous Tasks

TL;DR

This paper introduces DexMulti, a sample-efficient method for multi-stage dexterous manipulation that decomposes demonstrations into skills, enabling robust real-world performance with minimal data across various objects and tasks.

Contribution

DexMulti presents a novel approach that decomposes demonstrations into object-centric skills and retrieves them based on current object states, improving data efficiency and generalization in dexterous manipulation.

Findings

Achieves 66% success rate with only 3-4 demonstrations per object.

Outperforms diffusion policy baselines by 2-3x in data efficiency.

Demonstrates robust generalization to new objects and spatial variations.

Abstract

Dexterous hands enable concurrent prehensile and nonprehensile manipulation, such as holding one object while interacting with another, a capability essential for everyday tasks yet underexplored in robotics. Learning such long-horizon, contact-rich multi-stage behaviors is challenging because demonstrations are expensive to collect and end-to-end policies require substantial data to generalize across varied object geometries and placements. We present DexMulti, a sample-efficient approach for real-world dexterous multi-task manipulation that decomposes demonstrations into object-centric skills with well-defined temporal boundaries. Rather than learning monolithic policies, our method retrieves demonstrated skills based on current object geometry, aligns them to the observed object state using an uncertainty-aware estimator that tracks centroid and yaw, and executes them via a retrieve-align-execute paradigm. We evaluate on three multi-stage tasks requiring concurrent manipulation (Grasp + Pull, Grasp + Open, and Grasp + Grasp) across two dexterous hands (Allegro and LEAP) in over 1,000 real-world trials. Our approach achieves an average success rate of 66% on training objects with only 3-4 demonstrations per object, outperforming diffusion policy baselines by 2-3x while requiring far fewer demonstrations. Results demonstrate robust generalization to held-out objects and spatial variations up to +/-25 cm.