Grasp to Act: Dexterous Grasping for Tool Use in Dynamic Settings

TL;DR

This paper introduces Grasp-to-Act, a hybrid system combining physics-based grasp optimization and reinforcement learning to enable dexterous robots to perform stable, dynamic tool use tasks in real-world settings.

Contribution

It presents a novel approach that integrates human-inspired grasp synthesis with adaptive control for robust manipulation under dynamic forces.

Findings

Achieves zero-shot sim-to-real transfer for five dynamic tasks

Reduces in-hand slip during manipulation

Outperforms baseline methods in task success rates

Abstract

Achieving robust grasping with dexterous hands remains challenging, especially when manipulation involves dynamic forces such as impacts, torques, and continuous resistance--situations common in real-world tool use. Existing methods largely optimize grasps for static geometric stability and often fail once external forces arise during manipulation. We present Grasp-to-Act, a hybrid system that combines physics-based grasp optimization with reinforcement-learning-based grasp adaptation to maintain stable grasps throughout functional manipulation tasks. Our method synthesizes robust grasp configurations informed by human demonstrations and employs an adaptive controller that residually issues joint corrections to prevent in-hand slip while tracking the object trajectory. Grasp-to-Act enables robust zero-shot sim-to-real transfer across five dynamic tool-use tasks--hammering, sawing, cutting, stirring, and scooping--consistently outperforming baselines. Across simulation and real-world hardware trials with a 16-DoF dexterous hand, our method reduces translational and rotational in-hand slip and achieves the highest task completion rates, demonstrating stable functional grasps under dynamic, contact-rich conditions.