HAGI++: Head-Assisted Gaze Imputation and Generation

TL;DR

HAGI++ is a novel multi-modal diffusion-based method that leverages head orientation sensors and transformer models to improve gaze data imputation and generation, outperforming existing methods in accuracy and realism.

Contribution

The paper introduces HAGI++, the first approach to incorporate head orientation sensors into gaze imputation using a transformer-based diffusion model, enhancing accuracy and enabling gaze generation with wearable device data.

Findings

HAGI++ outperforms traditional interpolation and deep learning baselines in gaze imputation.

Gaze velocity distributions from HAGI++ closely match actual human gaze behavior.

Incorporating wrist motion data allows for effective gaze generation even with complete missing data.

Abstract

Mobile eye tracking plays a vital role in capturing human visual attention across both real-world and extended reality (XR) environments, making it an essential tool for applications ranging from behavioural research to human-computer interaction. However, missing values due to blinks, pupil detection errors, or illumination changes pose significant challenges for further gaze data analysis. To address this challenge, we introduce HAGI++ - a multi-modal diffusion-based approach for gaze data imputation that, for the first time, uses the integrated head orientation sensors to exploit the inherent correlation between head and eye movements. HAGI++ employs a transformer-based diffusion model to learn cross-modal dependencies between eye and head representations and can be readily extended to incorporate additional body movements. Extensive evaluations on the large-scale Nymeria, Ego-Exo4D, and HOT3D datasets demonstrate that HAGI++ consistently outperforms conventional interpolation methods and deep learning-based time-series imputation baselines in gaze imputation. Furthermore, statistical analyses confirm that HAGI++ produces gaze velocity distributions that closely match actual human gaze behaviour, ensuring more realistic gaze imputations. Moreover, by incorporating wrist motion captured from commercial wearable devices, HAGI++ surpasses prior methods that rely on full-body motion capture in the extreme case of 100% missing gaze data (pure gaze generation). Our method paves the way for more complete and accurate eye gaze recordings in real-world settings and has significant potential for enhancing gaze-based analysis and interaction across various application domains.