ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation

TL;DR

ARGaze introduces an autoregressive transformer model for online egocentric gaze estimation, leveraging temporal continuity and recent gaze history to improve prediction accuracy in first-person videos.

Contribution

The paper presents a novel autoregressive transformer approach for online egocentric gaze estimation, emphasizing sequential prediction with bounded gaze history for improved robustness.

Findings

Achieves state-of-the-art performance on egocentric benchmarks.

Autoregressive modeling with recent gaze history is crucial for robustness.

Enables bounded-resource streaming inference for real-time applications.

Abstract

Online egocentric gaze estimation predicts where a camera wearer is looking from first-person video using only past and current frames, a task essential for augmented reality and assistive technologies. Unlike third-person gaze estimation, this setting lacks explicit head or eye signals, requiring models to infer current visual attention from sparse, indirect cues such as hand-object interactions and salient scene content. We observe that gaze exhibits strong temporal continuity during goal-directed activities: knowing where a person looked recently provides a powerful prior for predicting where they look next. Inspired by vision-conditioned autoregressive decoding in vision-language models, we propose ARGaze, which reformulates gaze estimation as sequential prediction: at each timestep, a transformer decoder predicts current gaze by conditioning on (i) current visual features and (ii) a fixed-length Gaze Context Window of recent gaze target estimates. This design enforces causality and enables bounded-resource streaming inference. We achieve state-of-the-art performance across multiple egocentric benchmarks under online evaluation, with extensive ablations validating that autoregressive modeling with bounded gaze history is critical for robust prediction. We will release our source code and pre-trained models.