Optimization-Based Eye Tracking using Deflectometric Information

TL;DR

This paper introduces a novel optimization-based eye tracking method that leverages pixel-dense deflectometric surface measurements and differentiable rendering to accurately estimate eye parameters, outperforming existing reflection-based techniques.

Contribution

It presents a new approach combining deflectometric measurements with inverse rendering and gradient descent for improved eye tracking accuracy in VR/AR/MR applications.

Findings

Achieved mean relative gaze errors below 0.45 degrees in real-world tests.

Demonstrated a 6X accuracy improvement over state-of-the-art reflection-based methods in simulation.

Method works with ordinary video frames without requiring specific patterns.

Abstract

Eye tracking is an important tool with a wide range of applications in Virtual, Augmented, and Mixed Reality (VR/AR/MR) technologies. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse point light sources, or image-based and exploit 2D features of the acquired eye image. In this work, we attempt to significantly improve reflection-based methods by utilizing pixel-dense deflectometric surface measurements in combination with optimization-based inverse rendering algorithms. Utilizing the known geometry of our deflectometric setup, we develop a differentiable rendering pipeline based on PyTorch3D that simulates a virtual eye under screen illumination. Eventually, we exploit the image-screen-correspondence information from the captured measurements to find the eye's rotation, translation, and shape parameters with our renderer via gradient descent. In general, our method does not require a specific pattern and can work with ordinary video frames of the main VR/AR/MR screen itself. We demonstrate real-world experiments with evaluated mean relative gaze errors below 0.45 degrees at a precision better than 0.11 degrees. Moreover, we show an improvement of 6X over a representative reflection-based state-of-the-art method in simulation.