Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models

TL;DR

Video-Bench provides a comprehensive evaluation framework and toolkit for assessing Video-LLMs across understanding, reasoning, and decision-making tasks, revealing current models' limitations in human-like video comprehension.

Contribution

The paper introduces Video-Bench, a new benchmark and toolkit for systematically evaluating Video-LLMs' capabilities across multiple levels and tasks.

Findings

Current Video-LLMs underperform in human-like comprehension.

Video-Bench covers diverse tasks for comprehensive evaluation.

Toolkit automates metric calculation and scoring.

Abstract

Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries. In pursuit of the ultimate goal of achieving artificial general intelligence, a truly intelligent Video-LLM model should not only see and understand the surroundings, but also possess human-level commonsense, and make well-informed decisions for the users. To guide the development of such a model, the establishment of a robust and comprehensive evaluation system becomes crucial. To this end, this paper proposes \textit{Video-Bench}, a new comprehensive benchmark along with a toolkit specifically designed for evaluating Video-LLMs. The benchmark comprises 10 meticulously crafted tasks, evaluating the capabilities of Video-LLMs across three distinct levels: Video-exclusive Understanding, Prior Knowledge-based Question-Answering, and Comprehension and Decision-making. In addition, we introduce an automatic toolkit tailored to process model outputs for various tasks, facilitating the calculation of metrics and generating convenient final scores. We evaluate 8 representative Video-LLMs using \textit{Video-Bench}. The findings reveal that current Video-LLMs still fall considerably short of achieving human-like comprehension and analysis of real-world videos, offering valuable insights for future research directions. The benchmark and toolkit are available at: \url{https://github.com/PKU-YuanGroup/Video-Bench}.