Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand

TL;DR

This paper introduces a hierarchical reinforcement learning framework enabling robotic hands to manipulate articulated tools like tweezers, improving dexterity and success rates in object grasping tasks through a two-policy system and affordance estimation.

Contribution

The work presents a novel hierarchical GCRL approach with an affordance encoder and heuristic policy to enhance articulated tool manipulation in robotic hands.

Findings

Achieved 70.8% success rate in real-world object grasping tasks.

Demonstrated effective manipulation of diverse tool configurations.

Validated the approach with real-world experiments.

Abstract

Manipulating articulated tools, such as tweezers or scissors, has rarely been explored in previous research. Unlike rigid tools, articulated tools change their shape dynamically, creating unique challenges for dexterous robotic hands. In this work, we present a hierarchical, goal-conditioned reinforcement learning (GCRL) framework to improve the manipulation capabilities of anthropomorphic robotic hands using articulated tools. Our framework comprises two policy layers: (1) a low-level policy that enables the dexterous hand to manipulate the tool into various configurations for objects of different sizes, and (2) a high-level policy that defines the tool's goal state and controls the robotic arm for object-picking tasks. We employ an encoder, trained on synthetic pointclouds, to estimate the tool's affordance states--specifically, how different tool configurations (e.g., tweezer opening angles) enable grasping of objects of varying sizes--from input point clouds, thereby enabling precise tool manipulation. We also utilize a privilege-informed heuristic policy to generate replay buffer, improving the training efficiency of the high-level policy. We validate our approach through real-world experiments, showing that the robot can effectively manipulate a tweezer-like tool to grasp objects of diverse shapes and sizes with a 70.8 % success rate. This study highlights the potential of RL to advance dexterous robotic manipulation of articulated tools.