RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation

TL;DR

This paper introduces RaycastGrasp, a gaze-based interface using wearable mixed reality to improve robotic object manipulation for individuals with mobility impairments, emphasizing natural interaction and high accuracy.

Contribution

It presents a novel egocentric, gaze-guided manipulation system leveraging MR headsets, pretrained vision models, and real-time intent recognition for improved accessibility.

Findings

Manipulation accuracy improved significantly

System latency was reduced

Intent and object recognition accuracy exceeded 88%

Abstract

Robotic manipulators are increasingly used to assist individuals with mobility impairments in object retrieval. However, the predominant joystick-based control interfaces can be challenging due to high precision requirements and unintuitive reference frames. Recent advances in human-robot interaction have explored alternative modalities, yet many solutions still rely on external screens or restrictive control schemes, limiting their intuitiveness and accessibility. To address these challenges, we present an egocentric, gaze-guided robotic manipulation interface that leverages a wearable Mixed Reality (MR) headset. Our system enables users to interact seamlessly with real-world objects using natural gaze fixation from a first-person perspective, while providing augmented visual cues to confirm intent and leveraging a pretrained vision model and robotic arm for intent recognition and object manipulation. Experimental results demonstrate that our approach significantly improves manipulation accuracy, reduces system latency, and achieves single-pass intention and object recognition accuracy greater than 88% across multiple real-world scenarios. These results demonstrate the system's effectiveness in enhancing intuitiveness and accessibility, underscoring its practical significance for assistive robotics applications.