StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

TL;DR

StreamingBench is a new benchmark that evaluates the real-time streaming video understanding capabilities of Multimodal Large Language Models, revealing significant gaps compared to human performance and guiding future improvements.

Contribution

This paper introduces StreamingBench, the first comprehensive benchmark for assessing streaming video understanding in MLLMs across multiple core aspects.

Findings

Most advanced MLLMs perform significantly below human-level in streaming scenarios.

Current models struggle with real-time visual, omni-source, and contextual understanding.

Benchmark includes 18 tasks with 900 videos and 4,500 QA pairs.

Abstract

The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding, and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.