Harms in Repurposing Real-World Sensory Cues for Mixed Reality: A Causal Perspective

TL;DR

This paper explores how repurposing real-world sensory cues in mixed reality can cause perceptual harms, using causal inference to better understand safety risks and guide safer MR design.

Contribution

It introduces a causal inference perspective to analyze perceptual harms in MR, highlighting safety risks from sensory cue manipulations.

Findings

Repurposing sensory cues can lead to perceptual misinterpretations.

Causal inference helps identify potential safety risks in MR environments.

Understanding harms can inform safer mixed reality design.

Abstract

The rise of Mixed Reality (MR) stimulates new interactive techniques that seamlessly blend the virtual and physical environments. Just as virtual content could be overlayed onto the physical world for providing adaptive user interfaces [5, 8], emergent techniques "repurpose" everyday environments and sensory cues to support the virtual content [7, 9, 13-15]. For instance, a strong wind gust in the real world, rather than being distracting to the virtual experience, can be mapped with trees swaying in MR to achieve a unifying experience [15], as shown in Figure 1. Such techniques introduce stronger immersion, but they also expose users to overlooked perceptual manipulations, where safety risks arise from misperception of real-world events. In this work, we apply a causal inference perspective to understand the harms of repurposing real-world sensory cues for MR. We argue that by viewing the MR experience as a causal inference process of interpreting cues arising from both the virtual and physical world, MR designers and researchers can gain a new lens to understand potential perceptual manipulation harms.