Point & Grasp: Flexible Selection of Out-of-Reach Objects Through Probabilistic Cue Integration

TL;DR

This paper presents a probabilistic cue integration framework for selecting out-of-reach objects in mixed reality, combining pointing and grasp cues to improve accuracy and robustness.

Contribution

It introduces a novel probabilistic cue integration approach and a new dataset for gesture-based intent inference in out-of-reach object selection.

Findings

Cue integration improves selection accuracy and speed.

The method outperforms single-cue baselines.

The dataset captures unique grasping patterns.

Abstract

Selecting out-of-reach objects is a fundamental task in mixed reality (MR). Existing methods rely on a single cue or deterministically fuse multiple cues, leading to performance degradation when the dominant cue becomes unreliable. In this work, we introduce a probabilistic cue integration framework that enables flexible combination of multiple user-generated cues for intent inference. Inspired by natural grasping behavior, we instantiate the framework with pointing direction and grasp gestures as a new interaction technique, Point&Grasp. To this end, we collect the Out-of-Reach Grasping (ORG) dataset to train a robust likelihood model of the gestural cue, which captures grasping patterns not present in existing in-reach datasets. User studies demonstrate that our selection method with cue integration not only improves accuracy and speed over single-cue baselines, but also remains practically effective compared to state-of-the-art methods across various sources of ambiguity. The dataset and code are available at https://github.com/drlxj/point-and-grasp.