Low Latency Gaze Tracking via Latent Optical Sensing

TL;DR

This paper introduces a passive optical gaze tracking system that encodes spatial information directly in the optical domain, enabling real-time, low-latency gaze estimation with reduced computational and energy costs.

Contribution

The authors develop a novel optical encoding approach combined with neural inference to achieve ultra-low latency gaze tracking without high-bandwidth image processing.

Findings

Achieves end-to-end latency of 3.4 ms, outperforming existing systems.

Maintains competitive gaze estimation accuracy with significantly lower latency.

Reduces computational and energy requirements compared to traditional camera-based methods.

Abstract

We present a real-time gaze tracking system that directly acquires task-relevant latent features using a fully passive optical encoder. Instead of forming and processing full-resolution images, our approach leverages a microlens array with a co-designed binary chromium mask to perform spatially multiplexed optical encoding, producing a compact set of measurements sufficient for gaze estimation. By integrating sensing and feature extraction in the optical domain, the proposed system eliminates the need for high-bandwidth image readout and substantially reduces computational overhead. The encoded measurements are captured by a 4 x 4 phototransistor array and mapped to gaze direction using a lightweight neural network. Our proof-of-concept prototype enables an end-to-end sensing-to-inference latency of 3.4 ms, outperforming published research systems. We demonstrate the effectiveness of our approach on both simulated and real-world data, achieving competitive gaze estimation accuracy while significantly improving latency and energy efficiency compared to conventional camera-based pipelines. This work highlights the potential of task-driven optical sensing for ultra-low-latency, computationally efficient human-computer interaction systems.