Visual Room 2.0: Seeing is Not Understanding for MLLMs

TL;DR

This paper introduces Visual Room 2.0, a hierarchical benchmark to evaluate perception and cognition in multi-modal large language models, revealing their perceptual strengths but limited cognitive understanding and the non-causal link between perception and cognition.

Contribution

The paper proposes a new hierarchical benchmark for perception-cognition alignment in MLLMs and provides empirical analysis across 17 tasks, highlighting the gap between perception and understanding.

Findings

MLLMs are better at perception than cognition.

Cognition does not depend causally on perception-based reasoning.

Cognition improves with model size, perception does not necessarily do so.

Abstract

Can multi-modal large language models (MLLMs) truly understand what they can see? Extending Searle's Chinese Room into the multi-modal domain, this paper proposes the Visual Room argument: MLLMs may describe every visual detail precisely yet fail to comprehend the underlying emotions and intentions, namely seeing is not understanding. Building on this, we introduce \textit{Visual Room} 2.0, a hierarchical benchmark for evaluating perception-cognition alignment of MLLMs. We model human perceptive and cognitive processes across three levels: low, middle, and high, covering 17 representative tasks. The perception component ranges from attribute recognition to scene understanding, while the cognition component extends from textual entailment to causal and social reasoning. The dataset contains 350 multi-modal samples, each with six progressive questions (2,100 in total) spanning perception to cognition. Evaluating 10 state-of-the-art (SoTA) MLLMs, we highlight three key findings: (1) MLLMs exhibit stronger perceptual competence than cognitive ability (8.0\%$\uparrow$); (2) cognition appears not causally dependent on perception-based reasoning; and (3) cognition scales with model size, but perception does not consistently improve with larger variants. This work operationalizes Seeing $\ne$ Understanding as a testable hypothesis, offering a new paradigm from perceptual processing to cognitive reasoning in MLLMs. Our dataset is available at https://huggingface.co/datasets/LHK2003/PCBench.