MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs

TL;DR

MVU-Eval introduces the first comprehensive benchmark to evaluate multi-video understanding in multimodal large language models, addressing a critical gap for real-world applications like autonomous driving and sports analytics.

Contribution

This work presents MVU-Eval, a novel benchmark with 1,824 question-answer pairs across nearly 5,000 videos, specifically designed to assess multi-video understanding in MLLMs, which was previously unaddressed.

Findings

Current MLLMs show significant performance gaps in multi-video understanding.

The benchmark reveals limitations in existing models' ability to handle multi-video tasks.

Evaluation highlights the need for improved multi-video reasoning capabilities.

Abstract

The advent of Multimodal Large Language Models (MLLMs) has expanded AI capabilities to visual modalities, yet existing evaluation benchmarks remain limited to single-video understanding, overlooking the critical need for multi-video understanding in real-world scenarios (e.g., sports analytics and autonomous driving). To address this significant gap, we introduce MVU-Eval, the first comprehensive benchmark for evaluating Multi-Video Understanding for MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos from diverse domains, addressing both fundamental perception tasks and high-order reasoning tasks. These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics. Through extensive evaluation of state-of-the-art open-source and closed-source models, we reveal significant performance discrepancies and limitations in current MLLMs' ability to perform understanding across multiple videos. The benchmark will be made publicly available to foster future research.