Int3DNet: Scene-Motion Cross Attention Network for 3D Intention Prediction in Mixed Reality

TL;DR

Int3DNet is a novel scene-aware network that predicts 3D human intention areas from scene geometry and motion cues, improving real-time interaction in Mixed Reality environments.

Contribution

The paper introduces a cross attention fusion approach for 3D intention prediction directly from scene and motion data, without explicit object perception.

Findings

Outperforms baselines on MoGaze and CIRCLE datasets

Maintains accuracy over time horizons up to 1500 ms

Enables seamless human-MR interaction through intention prediction

Abstract

We propose Int3DNet, a scene-aware network that predicts 3D intention areas directly from scene geometry and head-hand motion cues, enabling robust human intention prediction without explicit object-level perception. In Mixed Reality (MR), intention prediction is critical as it enables the system to anticipate user actions and respond proactively, reducing interaction delays and ensuring seamless user experiences. Our method employs a cross attention fusion of sparse motion cues and scene point clouds, offering a novel approach that directly interprets the user's spatial intention within the scene. We evaluated Int3DNet on MoGaze and CIRCLE datasets, which are public datasets for full-body human-scene interactions, showing consistent performance across time horizons of up to 1500 ms and outperforming the baselines, even in diverse and unseen scenes. Moreover, we demonstrate the usability of proposed method through a demonstration of efficient visual question answering (VQA) based on intention areas. Int3DNet provides reliable 3D intention areas derived from head-hand motion and scene geometry, thus enabling seamless interaction between humans and MR systems through proactive processing of intention areas.