TeleFMG: A Wearable Force-Myography Device for Natural Teleoperation of Multi-finger Robotic Hands

TL;DR

This paper introduces TeleFMG, a wearable device that enables natural teleoperation of multi-finger robotic hands by translating forearm muscle signals into hand gestures using a neural network model.

Contribution

The paper presents a low-cost wearable FMG device and a spatio-temporal neural network model for intuitive teleoperation of robotic hands, with successful transfer learning capabilities.

Findings

Robust control of robotic hand through natural finger motions.

Effective transfer learning with limited new user data.

Successful object grasping and gesture mimicry by the robot.

Abstract

Teleoperation enables a user to perform dangerous tasks (e.g., work in disaster zones or in chemical plants) from a remote location. Nevertheless, common approaches often provide cumbersome and unnatural usage. In this letter, we propose TeleFMG, an approach for teleoperation of a multi-finger robotic hand through natural motions of the user's hand. By using a low-cost wearable Force-Myography (FMG) device, musculoskeletal activities on the user's forearm are mapped to hand poses which, in turn, are mimicked by a robotic hand. The mapping is performed by a spatio-temporal data-based model based on the Temporal Convolutional Network. The model considers spatial positions of the sensors on the forearm along with temporal dependencies of the FMG signals. A set of experiments show the ability of a teleoperator to control a multi-finger hand through intuitive and natural finger motion. A robot is shown to successfully mimic the user's hand in object grasping and gestures. Furthermore, transfer to a new user is evaluated while showing that fine-tuning with a limited amount of new data significantly improves accuracy.