Enhancing 3D Gaze Estimation in the Wild using Weak Supervision with Gaze Following Labels

TL;DR

This paper introduces a novel weakly-supervised framework and a modality-agnostic transformer architecture for improved 3D gaze estimation in real-world environments, leveraging diverse datasets and pseudo-labels.

Contribution

The paper proposes ST-WSGE and Gaze Transformer, enabling better generalization and cross-modal performance in 3D gaze estimation using weak supervision and combined image/video data.

Findings

Achieved state-of-the-art results on Gaze360 and GFIE datasets.

Demonstrated superior cross-domain performance over existing methods.

Enhanced video gaze estimation accuracy with cross-modal data.

Abstract

Accurate 3D gaze estimation in unconstrained real-world environments remains a significant challenge due to variations in appearance, head pose, occlusion, and the limited availability of in-the-wild 3D gaze datasets. To address these challenges, we introduce a novel Self-Training Weakly-Supervised Gaze Estimation framework (ST-WSGE). This two-stage learning framework leverages diverse 2D gaze datasets, such as gaze-following data, which offer rich variations in appearances, natural scenes, and gaze distributions, and proposes an approach to generate 3D pseudo-labels and enhance model generalization. Furthermore, traditional modality-specific models, designed separately for images or videos, limit the effective use of available training data. To overcome this, we propose the Gaze Transformer (GaT), a modality-agnostic architecture capable of simultaneously learning static and dynamic gaze information from both image and video datasets. By combining 3D video datasets with 2D gaze target labels from gaze following tasks, our approach achieves the following key contributions: (i) Significant state-of-the-art improvements in within-domain and cross-domain generalization on unconstrained benchmarks like Gaze360 and GFIE, with notable cross-modal gains in video gaze estimation; (ii) Superior cross-domain performance on datasets such as MPIIFaceGaze and Gaze360 compared to frontal face methods. Code and pre-trained models will be released to the community.