VideoScaffold: Elastic-Scale Visual Hierarchies for Streaming Video Understanding in MLLMs

TL;DR

VideoScaffold introduces a dynamic, hierarchical framework for streaming video understanding that adaptively refines event boundaries and aggregates semantics, enabling continuous, coherent comprehension with state-of-the-art results.

Contribution

It proposes a novel elastic-scale, hierarchical approach with EES and HEC components for real-time video understanding, addressing limitations of static methods.

Findings

Achieves state-of-the-art performance on streaming video benchmarks.

Seamlessly extends image-based MLLMs to continuous video comprehension.

Demonstrates effectiveness in both offline and streaming settings.

Abstract

Understanding long videos with multimodal large language models (MLLMs) remains challenging due to the heavy redundancy across frames and the need for temporally coherent representations. Existing static strategies, such as sparse sampling, frame compression, and clustering, are optimized for offline settings and often produce fragmented or over-compressed outputs when applied to continuous video streams. We present VideoScaffold, a dynamic representation framework designed for streaming video understanding. It adaptively adjusts event granularity according to video duration while preserving fine-grained visual semantics. VideoScaffold introduces two key components: Elastic-Scale Event Segmentation (EES), which performs prediction-guided segmentation to dynamically refine event boundaries, and Hierarchical Event Consolidation (HEC), which progressively aggregates semantically related segments into multi-level abstractions. Working in concert, EES and HEC enable VideoScaffold to transition smoothly from fine-grained frame understanding to abstract event reasoning as the video stream unfolds. Extensive experiments across both offline and streaming video understanding benchmarks demonstrate that VideoScaffold achieves state-of-the-art performance. The framework is modular and plug-and-play, seamlessly extending existing image-based MLLMs to continuous video comprehension. The code is available at https://github.com/zheng980629/VideoScaffold.