Neuromorphic Eye Tracking for Low-Latency Pupil Detection

TL;DR

This paper introduces a neuromorphic eye-tracking model using event-based sensors and spiking neural networks, achieving high accuracy with significantly reduced power and latency suitable for wearable AR/VR devices.

Contribution

It presents a novel neuromorphic adaptation of top eye-tracking models, replacing complex modules with lightweight layers to enhance efficiency and real-time performance.

Findings

Achieves 3.7-4.1px mean error, close to neuromorphic Retina system.

Reduces model size by 20x and compute by 850x compared to similar ANNs.

Projected to operate at 3.9-4.9 mW with 3 ms latency at 1 kHz.

Abstract

Eye tracking for wearable systems demands low latency and milliwatt-level power, but conventional frame-based pipelines struggle with motion blur, high compute cost, and limited temporal resolution. Such capabilities are vital for enabling seamless and responsive interaction in emerging technologies like augmented reality (AR) and virtual reality (VR), where understanding user gaze is key to immersion and interface design. Neuromorphic sensors and spiking neural networks (SNNs) offer a promising alternative, yet existing SNN approaches are either too specialized or fall short of the performance of modern ANN architectures. This paper presents a neuromorphic version of top-performing event-based eye-tracking models, replacing their recurrent and attention modules with lightweight LIF layers and exploiting depth-wise separable convolutions to reduce model complexity. Our models obtain 3.7-4.1px mean error, approaching the accuracy of the application-specific neuromorphic system, Retina (3.24px), while reducing model size by 20x and theoretical compute by 850x, compared to the closest ANN variant of the proposed model. These efficient variants are projected to operate at an estimated 3.9-4.9 mW with 3 ms latency at 1 kHz. The present results indicate that high-performing event-based eye-tracking architectures can be redesigned as SNNs with substantial efficiency gains, while retaining accuracy suitable for real-time wearable deployment.