Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark

TL;DR

This empirical study evaluates the reasoning capabilities of the Veo-3 video model across multiple dimensions using the MME-CoF benchmark, revealing strengths in short-term spatial coherence but limitations in long-term causal reasoning.

Contribution

The paper introduces the MME-CoF benchmark for standardized assessment of video models' reasoning and provides a comprehensive analysis of Veo-3's reasoning abilities and failure modes.

Findings

Veo-3 shows promising short-term spatial reasoning.

Limited in long-horizon causal and geometric reasoning.

Not yet reliable as standalone zero-shot reasoners.

Abstract

Recent video generation models can produce high-fidelity, temporally coherent videos, indicating that they may encode substantial world knowledge. Beyond realistic synthesis, they also exhibit emerging behaviors indicative of visual perception, modeling, and manipulation. Yet, an important question still remains: Are video models ready to serve as zero-shot reasoners in challenging visual reasoning scenarios? In this work, we conduct an empirical study to comprehensively investigate this question, focusing on the leading and popular Veo-3. We evaluate its reasoning behavior across 12 dimensions, including spatial, geometric, physical, temporal, and embodied logic, systematically characterizing both its strengths and failure modes. To standardize this study, we curate the evaluation data into MME-CoF, a compact benchmark that enables in-depth and thorough assessment of Chain-of-Frame (CoF) reasoning. Our findings reveal that while current video models demonstrate promising reasoning patterns on short-horizon spatial coherence, fine-grained grounding, and locally consistent dynamics, they remain limited in long-horizon causal reasoning, strict geometric constraints, and abstract logic. Overall, they are not yet reliable as standalone zero-shot reasoners, but exhibit encouraging signs as complementary visual engines alongside dedicated reasoning models. Project page: https://video-cof.github.io