HannesImitation: Grasping with the Hannes Prosthetic Hand via Imitation Learning

TL;DR

This paper introduces HannesImitation, a novel imitation learning approach for controlling the Hannes prosthetic hand, enabling effective grasping in unstructured environments using a new dataset and a diffusion policy that outperforms traditional methods.

Contribution

It presents a new imitation learning-based control policy for prosthetic hands, along with a dataset of grasping demonstrations and a diffusion model that improves grasping performance.

Findings

Successful grasping across diverse objects and scenarios

The diffusion policy outperforms segmentation-based visual servo control

Demonstrated effectiveness in unstructured environments

Abstract

Recent advancements in control of prosthetic hands have focused on increasing autonomy through the use of cameras and other sensory inputs. These systems aim to reduce the cognitive load on the user by automatically controlling certain degrees of freedom. In robotics, imitation learning has emerged as a promising approach for learning grasping and complex manipulation tasks while simplifying data collection. Its application to the control of prosthetic hands remains, however, largely unexplored. Bridging this gap could enhance dexterity restoration and enable prosthetic devices to operate in more unconstrained scenarios, where tasks are learned from demonstrations rather than relying on manually annotated sequences. To this end, we present HannesImitationPolicy, an imitation learning-based method to control the Hannes prosthetic hand, enabling object grasping in unstructured environments. Moreover, we introduce the HannesImitationDataset comprising grasping demonstrations in table, shelf, and human-to-prosthesis handover scenarios. We leverage such data to train a single diffusion policy and deploy it on the prosthetic hand to predict the wrist orientation and hand closure for grasping. Experimental evaluation demonstrates successful grasps across diverse objects and conditions. Finally, we show that the policy outperforms a segmentation-based visual servo controller in unstructured scenarios. Additional material is provided on our project page: https://hsp-iit.github.io/HannesImitation