Dynamic X-Ray Vision in Mixed Reality

TL;DR

This paper presents a real-time, gaze-responsive X-ray vision system for mixed reality headsets, enabling users to see through obstacles dynamically based on their position and gaze direction.

Contribution

It introduces a novel dynamic X-ray vision window for MR devices that adapts in real-time to user movement and gaze, integrating scene understanding, eye tracking, and clipping techniques.

Findings

Real-time rendering of X-ray vision window based on user position.

Dynamic adjustment of window location and transparency with eye gaze.

Implementation on HoloLens 2 demonstrating practical MR enhancement.

Abstract

X-ray vision, a technique that allows users to see through walls and other obstacles, is a popular technique for Augmented Reality (AR) and Mixed Reality (MR). In this paper, we demonstrate a dynamic X-ray vision window that is rendered in real-time based on the user's current position and changes with movement in the physical environment. Moreover, the location and transparency of the window are also dynamically rendered based on the user's eye gaze. We build this X-ray vision window for a current state-of-the-art MR Head-Mounted Device (HMD) -- HoloLens 2 by integrating several different features: scene understanding, eye tracking, and clipping primitive.