How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs

TL;DR

This paper introduces CVRR-ES, a comprehensive benchmark for evaluating Video-LMMs' reasoning and robustness in complex, real-world videos, revealing current models' limitations and proposing a prompting technique to improve performance.

Contribution

The paper presents a new evaluation suite for Video-LMMs that assesses reasoning and robustness across diverse real-world videos, and introduces a training-free prompting method to enhance existing models.

Findings

Most Video-LMMs struggle with robustness and reasoning in complex videos.

Open-source models perform worse than closed-source ones in the benchmark.

The proposed DSCP technique improves model performance without additional training.

Abstract

Recent advancements in Large Language Models (LLMs) have led to the development of Video Large Multi-modal Models (Video-LMMs) that can handle a wide range of video understanding tasks. These models have the potential to be deployed in real-world applications such as robotics, AI assistants, medical surgery, and autonomous vehicles. The widespread adoption of Video-LMMs in our daily lives underscores the importance of ensuring and evaluating their robust performance in mirroring human-like reasoning and interaction capabilities in complex, real-world contexts. However, existing benchmarks for Video-LMMs primarily focus on general video comprehension abilities and neglect assessing their reasoning capabilities over complex videos in the real-world context, and robustness of these models through the lens of user prompts as text queries. In this paper, we present the Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES), a novel benchmark that comprehensively assesses the performance of Video-LMMs across 11 diverse real-world video dimensions. We evaluate 9 recent models, including both open-source and closed-source variants, and find that most of the Video-LMMs, especially open-source ones, struggle with robustness and reasoning when dealing with complex videos. Based on our analysis, we develop a training-free Dual-Step Contextual Prompting (DSCP) technique to enhance the performance of existing Video-LMMs. Our findings provide valuable insights for building the next generation of human-centric AI systems with advanced robustness and reasoning capabilities. Our dataset and code are publicly available at: https://mbzuai-oryx.github.io/CVRR-Evaluation-Suite/.