Grasp Prediction based on Local Finger Motion Dynamics

TL;DR

This study demonstrates that real-time prediction of grasp points and object distance using hand motion data is feasible with high precision, enabling improved interactive experiences in mixed-reality environments.

Contribution

The paper introduces a method for predicting grasp timing and object distance from hand kinematics with high accuracy using simple LSTM networks.

Findings

Grasp timing can be predicted within 21 ms.

Object distance can be estimated within 1 cm.

Object size can be identified with over 97% accuracy.

Abstract

The ability to predict the object the user intends to grasp offers essential contextual information and may help to leverage the effects of point-to-point latency in interactive environments. This paper explores the feasibility and accuracy of real-time recognition of uninstrumented objects based on hand kinematics during reach-to-grasp actions. In a data collection study, we recorded the hand motions of 16 participants while reaching out to grasp and then moving real and synthetic objects. Our results demonstrate that even a simple LSTM network can predict the time point at which the user grasps an object with a precision better than 21 ms and the current distance to this object with a precision better than 1 cm. The target's size can be determined in advance with an accuracy better than 97%. Our results have implications for designing adaptive and fine-grained interactive user interfaces in ubiquitous and mixed-reality environments.