HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

TL;DR

HANDFUL introduces a resource-aware grasping framework for sequential dexterous manipulation, improving task success by conserving fingers for future actions, validated in simulation and on a real robot.

Contribution

This work presents HANDFUL, a novel learning framework and benchmark for resource-aware grasp planning in sequential manipulation tasks.

Findings

Prioritizing resource-aware grasps enhances second-subtask success.

Resource-aware grasping improves robustness over greedy initial grasp strategies.

The approach is validated on a real dexterous hand, demonstrating practical effectiveness.

Abstract

Dexterous robot hands offer rich opportunities for multifunctional manipulation, where a robot must execute multiple skills in sequence while maintaining control over previously grasped objects. Most prior work in dexterous manipulation focuses on single-object, single-skill tasks. In contrast, our insight is that many sequential tasks require resource-aware grasps that conserve fingers for future actions. In this paper, we study sequential grasp-conditioned dexterous manipulation, where a robot first grasps an object and then performs a second, distinct manipulation subtask while preserving the initial grasp. We introduce HANDFUL, a learning framework that models finger usage as a limited resource and encourages exploration of resource-aware grasps through finger-level contact rewards. These grasps are subsequently selected for downstream tasks via curriculum-based policy learning. We further propose HANDFUL-Bench, a simulation benchmark that introduces sequential dexterous manipulation tasks across multiple secondsubtask objectives, including pushing, pulling, and pressing, under a shared grasp-conditioned setup. Extensive simulation results demonstrate that prioritizing resource-aware grasps improves second-subtask success and robustness compared to a baseline that greedily optimizes the initial grasp before attempting the second subtask. We additionally validate our approach on a real dexterous LEAP hand. Together, this work establishes resource-aware grasp planning as a key principle for multifunctional dexterous manipulation. Supplementary material is available on our website: https://handful-dex.github.io.