Semantic Reality: Interactive Context-Aware Visualization of Inter-Object Relationships in Augmented Reality

TL;DR

Semantic Reality is an AR system that visualizes and interacts with inter-object relationships to improve understanding and task performance in augmented reality environments.

Contribution

It introduces a connectivity-centered interaction paradigm and system architecture for persistent, interactive inter-object relationship visualization in AR.

Findings

Participants reported clearer understanding of object relationships.

Participants experienced higher engagement and satisfaction.

Connectivity visualization aided planning and disambiguation.

Abstract

Bridging the physical and digital world through interaction remains a core challenge in augmented reality (AR). Existing systems target single objects, limiting support for planning, comparison, and assembly tasks that depend on relationships among multiple items. We present Semantic Reality, an AR system focused on surfacing inter-object connectivity and making it interactive. Leveraging multimodal reasoning, spatial anchoring, and physical action recognition, Semantic Reality maintains a persistent model of objects around the user and their relationships. Connections are visualized in-situ to highlight compatibility, reveal next steps, and reduce ambiguity during tasks. We contribute a connectivity-centered interaction paradigm and a system architecture that couples anchor tracking, action sensing, and model inference to construct a live connectivity graph. In an exploratory study comparing Semantic Reality to a single-object baseline, participants reported clearer inter-object understanding and higher engagement and satisfaction, without increased workload. A scenario study illustrates where connectivity aids planning, sequencing, and disambiguation.