World Mouse: Exploring Interactions with a Cross-Reality Cursor

TL;DR

World Mouse introduces a cross-reality cursor that adapts 2D mouse interactions for complex 3D environments by leveraging surface normals and semantic segmentation, enabling seamless real and virtual interactions.

Contribution

It presents a novel cross-reality cursor system that combines surface-based interaction and semantic understanding to improve XR interactions over previous virtual-only methods.

Findings

Enables precise object manipulation in XR environments

Facilitates seamless navigation between real and virtual spaces

Demonstrates improved interaction accuracy with prototypes

Abstract

As Extended Reality (XR) systems increasingly map and understand the physical world, interacting with these blended representations remains challenging. The current push for "natural" inputs has its trade-offs: touch is limited by human reach and fatigue, while gaze often lacks the precision for fine interaction. To bridge this gap, we introduce World Mouse, a cross-reality cursor that reinterprets the familiar 2D desktop mouse for complex 3D scenes. The system is driven by two core mechanisms: within-object interaction, which uses surface normals for precise cursor placement, and between-object navigation, which leverages interpolation to traverse empty space. Unlike previous virtual-only approaches, World Mouse leverages semantic segmentation and mesh reconstruction to treat physical objects as interactive surfaces. Through a series of prototypes, including object manipulation and screen-to-world transitions, we illustrate how cross-reality cursors may enable seamless interactions across real and virtual environments.