Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations

TL;DR

Reality Proxy introduces abstract representations of real-world objects in MR, enabling easier, more versatile interactions by decoupling physical constraints from user input, enhanced with AI for semantic understanding.

Contribution

The paper presents a novel proxy-based interaction system in MR that leverages AI for semantic enrichment, allowing complex interactions without new gestures or menus.

Findings

Enables multi-object selection and filtering in MR

Improves interaction with occluded or distant objects

Expert evaluation confirms system utility and usability

Abstract

Interacting with real-world objects in Mixed Reality (MR) often proves difficult when they are crowded, distant, or partially occluded, hindering straightforward selection and manipulation. We observe that these difficulties stem from performing interaction directly on physical objects, where input is tightly coupled to their physical constraints. Our key insight is to decouple interaction from these constraints by introducing proxies-abstract representations of real-world objects. We embody this concept in Reality Proxy, a system that seamlessly shifts interaction targets from physical objects to their proxies during selection. Beyond facilitating basic selection, Reality Proxy uses AI to enrich proxies with semantic attributes and hierarchical spatial relationships of their corresponding physical objects, enabling novel and previously cumbersome interactions in MR - such as skimming, attribute-based filtering, navigating nested groups, and complex multi object selections - all without requiring new gestures or menu systems. We demonstrate Reality Proxy's versatility across diverse scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. An expert evaluation suggests the system's utility and usability, suggesting that proxy-based abstractions offer a powerful and generalizable interaction paradigm for future MR systems.