From Hand-Perspective Visual Information to Grasp Type Probabilities: Deep Learning via Ranking Labels

TL;DR

This paper introduces a probabilistic ranking model using deep learning to predict grasp type probabilities for prosthetic hands, leveraging eye- and hand-view visual data and human-in-the-loop feedback for improved control.

Contribution

It proposes a novel ranking-based probabilistic classifier using the Plackett-Luce model for grasp prediction, integrating multi-view visual data and human-in-the-loop feedback.

Findings

Effective prediction of grasp probabilities from visual data.

Integration of ranking labels improves control accuracy.

Model applicable to standard CNN frameworks.

Abstract

Limb deficiency severely affects the daily lives of amputees and drives efforts to provide functional robotic prosthetic hands to compensate this deprivation. Convolutional neural network-based computer vision control of the prosthetic hand has received increased attention as a method to replace or complement physiological signals due to its reliability by training visual information to predict the hand gesture. Mounting a camera into the palm of a prosthetic hand is proved to be a promising approach to collect visual data. However, the grasp type labelled from the eye and hand perspective may differ as object shapes are not always symmetric. Thus, to represent this difference in a realistic way, we employed a dataset containing synchronous images from eye- and hand- view, where the hand-perspective images are used for training while the eye-view images are only for manual labelling. Electromyogram (EMG) activity and movement kinematics data from the upper arm are also collected for multi-modal information fusion in future work. Moreover, in order to include human-in-the-loop control and combine the computer vision with physiological signal inputs, instead of making absolute positive or negative predictions, we build a novel probabilistic classifier according to the Plackett-Luce model. To predict the probability distribution over grasps, we exploit the statistical model over label rankings to solve the permutation domain problems via a maximum likelihood estimation, utilizing the manually ranked lists of grasps as a new form of label. We indicate that the proposed model is applicable to the most popular and productive convolutional neural network frameworks.