Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

TL;DR

This paper introduces a synthetic training pipeline for eye-in-hand grasp classification on prosthetic hands, reducing data collection efforts and improving generalization, demonstrated on the Hannes prosthesis with publicly available code.

Contribution

It presents a novel synthetic data generation pipeline for hand pre-shape classification supporting multiple grasp types, enhancing generalization and practical deployment on prosthetic devices.

Findings

Synthetic training improves grasp classification accuracy.

Models trained on synthetic data outperform those trained on real data.

The approach is validated on the Hannes prosthetic hand.

Abstract

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-in-hand systems automatically control the hand pre-shaping before the grasp, based on visual input coming from a camera on the wrist. In this paper, we present an eye-in-hand learning-based approach for hand pre-shape classification from RGB sequences. Differently from previous work, we design the system to support the possibility to grasp each considered object part with a different grasp type. In order to overcome the lack of data of this kind and reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. We make publicly available the code and dataset to reproduce the presented results.