StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding

TL;DR

StreamAgent introduces anticipatory, goal-driven video understanding by predicting future events and regions, enabling more responsive and efficient real-time analysis in streaming videos.

Contribution

The paper presents a novel anticipatory agent with a hierarchical memory mechanism for proactive streaming video understanding, outperforming existing methods.

Findings

Outperforms existing methods in response accuracy.

Enhances real-time efficiency in streaming video tasks.

Effectively anticipates future events and regions of interest.

Abstract

Real-time streaming video understanding in domains such as autonomous driving and intelligent surveillance poses challenges beyond conventional offline video processing, requiring continuous perception, proactive decision making, and responsive interaction based on dynamically evolving visual content. However, existing methods rely on alternating perception-reaction or asynchronous triggers, lacking task-driven planning and future anticipation, which limits their real-time responsiveness and proactive decision making in evolving video streams. To this end, we propose a StreamAgent that anticipates the temporal intervals and spatial regions expected to contain future task-relevant information to enable proactive and goal-driven responses. Specifically, we integrate question semantics and historical observations through prompting the anticipatory agent to anticipate the temporal progression of key events, align current observations with the expected future evidence, and subsequently adjust the perception action (e.g., attending to task-relevant regions or continuously tracking in subsequent frames). To enable efficient inference, we design a streaming KV-cache memory mechanism that constructs a hierarchical memory structure for selective recall of relevant tokens, enabling efficient semantic retrieval while reducing the overhead of storing all tokens in the traditional KV-cache. Extensive experiments on streaming and long video understanding tasks demonstrate that our method outperforms existing methods in response accuracy and real-time efficiency, highlighting its practical value for real-world streaming scenarios.