StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

TL;DR

StreamGaze introduces a new benchmark to evaluate how well Multimodal Large Language Models utilize gaze signals for temporal reasoning and proactive understanding in streaming videos, highlighting current limitations.

Contribution

This work presents the first benchmark specifically designed to assess gaze-guided reasoning in streaming video understanding with MLLMs, including a novel QA generation pipeline.

Findings

MLLMs lag behind humans in gaze-based streaming video tasks.

Gaze-guided reasoning reveals key limitations in current models.

Analysis suggests directions for improving gaze utilization in models.

Abstract

Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. These tasks evaluate whether models can use real-time gaze signals to follow shifting attention and infer user intentions based only on past and currently observed frames. To build StreamGaze, we develop a gaze-video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support continued research in gaze-guided streaming video understanding.