How Far Are Video Models from True Multimodal Reasoning?

TL;DR

This paper introduces CLVG-Bench, a comprehensive evaluation framework for assessing the true multimodal reasoning capabilities of video models, revealing significant gaps in current state-of-the-art models.

Contribution

The paper presents CLVG-Bench and AVE, novel tools for rigorous, scalable, and human-aligned assessment of video models' reasoning abilities across complex scenarios.

Findings

State-of-the-art models perform poorly on logical and interactive tasks (<25% success)

Current models excel only in basic understanding and reasoning subtasks

The framework exposes critical bottlenecks in multimodal reasoning and physical grounding.

Abstract

Despite remarkable progress toward general-purpose video models, a critical question remains unanswered: how far are these models from achieving true multimodal reasoning? Existing benchmarks fail to address this question rigorously, as they remain constrained by straightforward task designs and fragmented evaluation metrics that neglect complex multimodal reasoning. To bridge this gap, we introduce CLVG-Bench, an evaluation framework designed to probe video models' zero-shot reasoning capabilities via Context Learning in Video Generation. CLVG-Bench comprises more than 1,000 high-quality, manually annotated metadata across 6 categories and 47 subcategories, covering complex scenarios including physical simulation, logical reasoning, and interactive contexts. To enable rigorous and scalable assessment, we further propose an Adaptive Video Evaluator (AVE) that aligns with human expert perception using minimal annotations, delivering interpretable textual feedback across diverse video context tasks. Extensive experiments reveal a striking answer to our central question: while state-of-the-art (SOTA) video models, such as Seedance 2.0, demonstrate competence on certain understanding and reasoning subtasks, they fall substantially short with logically grounded and interactive generation tasks (achieving success rates <25% and ~0%, respectively), exposing multimodal reasoning and physical grounding as critical bottlenecks. By systematically quantifying these limitations, the proposed method provides actionable feedbacks and a clear roadmap toward truly robust, general-purpose video models. CLVG-Bench and code are released here.