Predicting User Grasp Intentions in Virtual Reality

TL;DR

This paper compares classification and regression models, especially LSTM networks, for predicting user grasp intentions in VR, finding regression approaches more robust and promising for enhancing immersive experiences.

Contribution

It introduces a comprehensive evaluation of time-series regression models, notably LSTM, for predicting VR user grasp intentions, highlighting their advantages over classification methods.

Findings

Regression models outperform classification in generalization across users.

LSTM-based models achieve timing errors within 0.25 seconds.

Distance prediction errors range from 5 to 20 cm in critical moments.

Abstract

Predicting user intentions in virtual reality (VR) is crucial for creating immersive experiences, particularly in tasks involving complex grasping motions where accurate haptic feedback is essential. In this work, we leverage time-series data from hand movements to evaluate both classification and regression approaches across 810 trials with varied object types, sizes, and manipulations. Our findings reveal that classification models struggle to generalize across users, leading to inconsistent performance. In contrast, regression-based approaches, particularly those using Long Short Term Memory (LSTM) networks, demonstrate more robust performance, with timing errors within 0.25 seconds and distance errors around 5-20 cm in the critical two-second window before a grasp. Despite these improvements, predicting precise hand postures remains challenging. Through a comprehensive analysis of user variability and model interpretability, we explore why certain models fail and how regression models better accommodate the dynamic and complex nature of user behavior in VR. Our results underscore the potential of machine learning models to enhance VR interactions, particularly through adaptive haptic feedback, and lay the groundwork for future advancements in real-time prediction of user actions in VR.