Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models

TL;DR

This paper introduces Think While Watching, a novel streaming video reasoning framework that maintains segment-level memory for improved multi-turn video understanding in multimodal large language models, enabling better online reasoning.

Contribution

It proposes a memory-anchored streaming reasoning method with a multi-stage training strategy and causality enforcement, enhancing multi-turn video reasoning in multimodal LLMs.

Findings

Improves single-round accuracy by 2.6% on StreamingBench.

Achieves 3.79% higher accuracy on OVO-Bench.

Reduces output tokens by 56% in multi-round settings.

Abstract

Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/