An Efficient Streaming Video Understanding Framework with Agentic Control

TL;DR

The paper introduces R3-Streaming, a dynamic control framework for streaming video understanding that optimizes memory, response readiness, and computation routing to improve efficiency and accuracy under latency constraints.

Contribution

It presents a novel cascaded control approach with age-aware memory compression and a reinforcement learning-based compute routing method for streaming video understanding.

Findings

Achieves state-of-the-art results on OVO-Bench and StreamingBench datasets.

Reduces visual token usage by 95 to 96 percent.

Effectively balances model complexity and latency in streaming video tasks.

Abstract

Streaming video requires handling dynamic information density under strict latency budgets. Yet, existing methods typically employ static strategies, such as fixed memory compression or reliance on a single model, forcing a trade-off: fast models fail on complex queries, while always-on heavy models violate real-time constraints and overcomplicate simple queries. Rather than fixing these decisions upfront, we propose R3-Streaming (Remember, Respond, Reason), which formulates streaming video understanding as a cascaded control problem: for each query, the system compresses memory, judges response readiness, and routes computation sequentially, so that each downstream decision builds on progressively refined information states. To optimize this pipeline, we introduce an age-aware forgetting policy for memory compression, as aggressively compressing historical frames can yield substantial performance gains. For compute routing, we propose TB-GRPO, a target-balanced reinforcement learning objective that routes hard queries to a stronger model while preventing mode collapse. Extensive evaluations demonstrate that R3-Streaming achieves state-of-the-art results among streaming MLLMs, reaching 57.92 on OVO-Bench and 76.36 on StreamingBench, while reducing visual token usage by 95 to 96 percent.