MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models

TL;DR

MM-CoT is a new benchmark designed to evaluate whether multimodal models genuinely ground their reasoning in visual evidence and maintain logical coherence, addressing limitations of existing generation-focused benchmarks.

Contribution

The paper introduces MM-CoT, a diagnostic benchmark that assesses visual grounding and logical coherence in multimodal models' chain-of-thought reasoning, filling a critical evaluation gap.

Findings

Leading models struggle with visual grounding and logical coherence.

MM-CoT reveals a gap between generative fluency and reasoning fidelity.

Benchmark correlates poorly with existing metrics, indicating it measures unique reasoning aspects.

Abstract

The ability to perform Chain-of-Thought (CoT) reasoning marks a major milestone for multimodal models (MMs), enabling them to solve complex visual reasoning problems. Yet a critical question remains: is such reasoning genuinely grounded in visual evidence and logically coherent? Existing benchmarks emphasize generation but neglect verification, i.e., the capacity to assess whether a reasoning chain is both visually consistent and logically valid. To fill this gap, we introduce MM-CoT, a diagnostic benchmark specifically designed to probe the visual grounding and logical coherence of CoT reasoning in MMs. Instead of generating free-form explanations, models must select the sole event chain that satisfies two orthogonal constraints: (i) visual consistency, ensuring all steps are anchored in observable evidence, and (ii) logical coherence, ensuring causal and commonsense validity. Adversarial distractors are engineered to violate one of these constraints, exposing distinct reasoning failures. We evaluate leading vision-language models on MM-CoT and find that even the most advanced systems struggle, revealing a sharp discrepancy between generative fluency and true reasoning fidelity. MM-CoT shows low correlation with existing benchmarks, confirming that it measures a unique combination of visual grounding and logical reasoning. This benchmark provides a foundation for developing future models that reason not just plausibly, but faithfully and coherently within the visual world.