3D Prior is All You Need: Cross-Task Few-shot 2D Gaze Estimation

TL;DR

This paper presents a novel framework that adapts pre-trained 3D gaze estimation models for accurate 2D gaze prediction on unseen devices using few-shot learning, bridging the gap between 3D and 2D gaze estimation.

Contribution

It introduces a physics-based differentiable projection module and a dynamic pseudo-labeling strategy to effectively transfer 3D gaze models to 2D gaze estimation with limited data.

Findings

Achieves superior performance on MPIIGaze, EVE, and GazeCapture datasets.

Effectively models screen poses and handles unknown device configurations.

Demonstrates strong potential for real-world gaze estimation applications.

Abstract

3D and 2D gaze estimation share the fundamental objective of capturing eye movements but are traditionally treated as two distinct research domains. In this paper, we introduce a novel cross-task few-shot 2D gaze estimation approach, aiming to adapt a pre-trained 3D gaze estimation network for 2D gaze prediction on unseen devices using only a few training images. This task is highly challenging due to the domain gap between 3D and 2D gaze, unknown screen poses, and limited training data. To address these challenges, we propose a novel framework that bridges the gap between 3D and 2D gaze. Our framework contains a physics-based differentiable projection module with learnable parameters to model screen poses and project 3D gaze into 2D gaze. The framework is fully differentiable and can integrate into existing 3D gaze networks without modifying their original architecture. Additionally, we introduce a dynamic pseudo-labelling strategy for flipped images, which is particularly challenging for 2D labels due to unknown screen poses. To overcome this, we reverse the projection process by converting 2D labels to 3D space, where flipping is performed. Notably, this 3D space is not aligned with the camera coordinate system, so we learn a dynamic transformation matrix to compensate for this misalignment. We evaluate our method on MPIIGaze, EVE, and GazeCapture datasets, collected respectively on laptops, desktop computers, and mobile devices. The superior performance highlights the effectiveness of our approach, and demonstrates its strong potential for real-world applications.