StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

TL;DR

StreamingEval introduces a comprehensive evaluation framework for assessing the performance and deployability of streaming video understanding models under realistic resource constraints, highlighting current gaps and guiding future research.

Contribution

It presents a unified protocol for benchmarking streaming Video-LLMs, considering efficiency, storage, and accuracy in a standardized manner.

Findings

Current models lag behind real-world streaming requirements.

Significant trade-offs exist between efficiency and accuracy.

Benchmarking reveals gaps in deployability of existing Video-LLMs.

Abstract

Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.