GlintMarkers: Spatial Perception on XR Eyewear using Corneal Reflections

TL;DR

GlintMarkers leverages corneal reflections captured by XR eyewear cameras to enable gaze-driven spatial perception, including object orientation, distance, and identification.

Contribution

It introduces a novel passive marker design and a PnP-based framework to extract spatial and semantic info from corneal reflections in XR devices.

Findings

Successfully estimates object orientation and distance using corneal reflections.

Enables object identification through unique marker patterns.

Demonstrates feasibility of gaze-driven spatial perception in XR using corneal imaging.

Abstract

We present GlintMarkers, the first system to perform gaze-driven spatial perception using the inward-facing cameras on XR eyewear. Our key observation is that the cornea acts as a mirror that encodes both gaze direction and visual information about the environment in a small, low-contrast reflection. To extract spatial and semantic information from this reflection despite the camera's limited pixel budget, we present a passive retroreflective marker design that concentrates reflected near-infrared light onto the cornea, producing bright glint patterns. We develop a custom Perspective-n-Point (PnP) estimation framework adapted to corneal imaging and perform orientation and distance estimation of tagged objects, as well as unique object identification.