Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark

TL;DR

Gen-ViRe is a new benchmark designed to evaluate the reasoning capabilities of video generation models in simulating real-world dynamics through multi-step, physics-grounded reasoning tasks, addressing a gap in existing evaluation methods.

Contribution

This paper introduces Gen-ViRe, the first comprehensive framework to quantitatively assess visual reasoning in video models across multiple cognitive dimensions and subtasks.

Findings

State-of-the-art models show high visual fidelity but limited reasoning depth.

Gen-ViRe provides diagnostic insights into model reasoning abilities.

Baseline results highlight significant room for improvement in visual reasoning.

Abstract

While Chain-of-Thought (CoT) prompting enables sophisticated symbolic reasoning in LLMs, it remains confined to discrete text and cannot simulate the continuous, physics-governed dynamics of the real world. Recent video generation models have emerged as potential world simulators through Chain-of-Frames (CoF) reasoning -- materializing thought as frame-by-frame visual sequences, with each frame representing a physically-grounded reasoning step. Despite compelling demonstrations, a challenge persists: existing benchmarks, focusing on fidelity or alignment, do not assess CoF reasoning and thus cannot measure core cognitive abilities in multi-step planning, algorithmic logic, or abstract pattern extrapolation. This evaluation void prevents systematic understanding of model capabilities and principled guidance for improvement. We introduce Gen-ViRe (Generative Visual Reasoning Benchmark), a framework grounded in cognitive science and real-world AI applications, which decomposes CoF reasoning into six cognitive dimensions -- from perceptual logic to abstract planning -- and 24 subtasks. Through multi-source data curation, minimal prompting protocols, and hybrid VLM-assisted evaluation with detailed criteria, Gen-ViRe delivers the first quantitative assessment of video models as reasoners. Our experiments on SOTA systems reveal substantial discrepancies between impressive visual quality and actual reasoning depth, establishing baselines and diagnostic tools to advance genuine world simulators.