GazeD: Context-Aware Diffusion for Accurate 3D Gaze Estimation

TL;DR

GazeD is a novel diffusion-based method that jointly estimates 3D gaze and human pose from a single RGB image, leveraging context and a new gaze representation for improved accuracy.

Contribution

It introduces GazeD, a diffusion model that jointly predicts 3D gaze and pose, using a novel gaze representation as a body joint, achieving state-of-the-art results.

Findings

Achieves state-of-the-art 3D gaze estimation performance

Outperforms methods using temporal information

Effective joint estimation of gaze and pose from a single image

Abstract

We introduce GazeD, a new 3D gaze estimation method that jointly provides 3D gaze and human pose from a single RGB image. Leveraging the ability of diffusion models to deal with uncertainty, it generates multiple plausible 3D gaze and pose hypotheses based on the 2D context information extracted from the input image. Specifically, we condition the denoising process on the 2D pose, the surroundings of the subject, and the context of the scene. With GazeD we also introduce a novel way of representing the 3D gaze by positioning it as an additional body joint at a fixed distance from the eyes. The rationale is that the gaze is usually closely related to the pose, and thus it can benefit from being jointly denoised during the diffusion process. Evaluations across three benchmark datasets demonstrate that GazeD achieves state-of-the-art performance in 3D gaze estimation, even surpassing methods that rely on temporal information. Project details will be available at https://aimagelab.ing.unimore.it/go/gazed.