RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video

TL;DR

RTV-Bench is a comprehensive benchmark designed to evaluate multimodal large language models on continuous, real-time video streams, highlighting current limitations and guiding future improvements in dynamic video understanding.

Contribution

We introduce RTV-Bench, a detailed benchmark with multi-timestamp questions and hierarchical structures to assess MLLMs on real-time video analysis, covering perception, understanding, and reasoning.

Findings

Real-time models outperform offline models but still lag behind proprietary systems.

Scaling model size improves performance, but increasing input frame density does not always help.

Current architectures face limitations in handling long-horizon video streams.

Abstract

Multimodal Large Language Models (MLLMs) have made rapid progress in perception, understanding, and reasoning, yet existing benchmarks fall short in evaluating these abilities under continuous and dynamic real-world video streams. Such settings require models to maintain coherent understanding and reasoning as visual scenes evolve over time. **We introduce RTV-Bench, a fine-grained benchmark for real-time video analysis with MLLMs**. It is built upon three key principles: multi-timestamp question answering, hierarchical question structures spanning perception and reasoning, and multi-dimensional evaluation of continuous perception, understanding, and reasoning. RTV-Bench comprises 552 diverse videos and 4,608 carefully curated QA pairs covering a wide range of dynamic scenarios. We evaluate a broad range of state-of-the-art MLLMs, including proprietary, open-source offline, and open-source real-time models. Our results show that real-time models generally outperform offline counterparts but still lag behind leading proprietary systems. While scaling model capacity generally yields performance gains, simply increasing the density of sampled input frames does not consistently translate into improved results. These observations suggest inherent limitations in current architectures when handling long-horizon video streams, underscoring the need for models explicitly designed for streaming video processing and analysis.