WAT: Online Video Understanding Needs Watching Before Thinking

TL;DR

WAT is a two-stage online video reasoning framework that efficiently processes streaming videos by separating watching and thinking, utilizing hierarchical memory and retrieval mechanisms, and is supported by a new dataset.

Contribution

The paper introduces WAT, a novel two-stage online video reasoning framework with hierarchical memory and retrieval, and presents WAT-85K, a dataset for training and evaluating online video understanding models.

Findings

WAT achieves 77.7% accuracy on StreamingBench.

WAT outperforms existing open-source online Video LLMs.

Operates at real-time frame rates.

Abstract

Multimodal Large Language Models (MLLMs) have shown strong capabilities in image understanding, motivating recent efforts to extend them to video reasoning. However, existing Video LLMs struggle in online streaming scenarios, where long temporal context must be preserved under strict memory constraints. We propose WAT (Watching Before Thinking), a two-stage framework for online video reasoning. WAT separates processing into a query-independent watching stage and a query-triggered thinking stage. The watching stage builds a hierarchical memory system with a Short-Term Memory (STM) that buffers recent frames and a fixed-capacity Long-Term Memory (LTM) that maintains a diverse summary of historical content using a redundancy-aware eviction policy. In the thinking stage, a context-aware retrieval mechanism combines the query with the current STM context to retrieve relevant historical frames from the LTM for cross-temporal reasoning. To support training for online video tasks, we introduce WAT-85K, a dataset containing streaming-style annotations emphasizing real-time perception, backward tracing, and forecasting. Experiments show that WAT achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench, outperforming existing open-source online Video LLMs while operating at real-time frame rates.