WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMs

TL;DR

WeaveTime enhances Video-LLMs for streaming video analysis by teaching models to understand temporal order and focus dynamically on relevant past information, improving accuracy and efficiency in online scenarios.

Contribution

It introduces a novel, model-agnostic framework with a lightweight training objective and dynamic focus mechanism for streaming Video-LLMs, addressing core limitations of time-agnosticism.

Findings

Improves accuracy on streaming video benchmarks

Reduces latency in video processing

Enhances temporal reasoning in Video-LLMs

Abstract

Recent advances in Multimodal Large Language Models have greatly improved visual understanding and reasoning, yet their quadratic attention and offline training protocols make them ill-suited for streaming settings where frames arrive sequentially and future observations are inaccessible. We diagnose a core limitation of current Video-LLMs, namely Time-Agnosticism, in which videos are treated as an unordered bag of evidence rather than a causally ordered sequence, yielding two failures in streams: temporal order ambiguity, in which the model cannot follow or reason over the correct chronological order, and past-current focus blindness where it fails to distinguish present observations from accumulated history. We present WeaveTime, a simple, efficient, and model agnostic framework that first teaches order and then uses order. We introduce a lightweight Temporal Reconstruction objective-our Streaming Order Perception enhancement-that instills order aware representations with minimal finetuning and no specialized streaming data. At inference, a Past-Current Dynamic Focus Cache performs uncertainty triggered, coarse-to-fine retrieval, expanding history only when needed. Plugged into exsiting Video-LLM without architectural changes, WeaveTime delivers consistent gains on representative streaming benchmarks, improving accuracy while reducing latency. These results establish WeaveTime as a practical path toward time aware stream Video-LLMs under strict online, time causal constraints. Code and weights will be made publicly available. Project Page: https://zhangyl4.github.io/publications/weavetime/